GPTBot, ClaudeBot & friends: managing AI crawlers without erasing your brand

In mid-2024, Cloudflare shipped a button that blocks all known AI bots in one click, noting the "meteoric" rise of generative-AI crawling and pitching the toggle to any customer uneasy about it.⁴ It's a good product answering real anger — much of it from publishers watching their work get ingested without payment.

But somewhere along the way, "block the AI bots" became a default posture that B2B marketing sites copy from publishers without asking the question that actually matters: what does each of these crawlers feed, and can we afford to be absent from it? If your business model is selling products rather than pageviews, the copied robots.txt block is often self-sabotage — you're not protecting inventory, you're deleting yourself from the shortlists your buyers now read.

So let's do this properly: the catalogue, the real cost of each block, and a decision framework.

The crawler catalogue: who's actually knocking

These are the user agents that matter most, per the operators' own documentation.¹²³ The critical column is the third one — what the crawler feeds:

Anthropic also documents lighter-weight agents (Claude-User for user-initiated fetches, Claude-SearchBot for search) alongside ClaudeBot, and the same feed-based logic applies to each.²

What blocking actually costs: three different clocks

Notice the pattern in the table — the crawlers sort into three tiers, each with a different feedback speed:

1. Live-browsing agents (ChatGPT-User, Claude-User). Cost is immediate and per-conversation. A buyer asks "compare Fathomly and Amplitude pricing," the assistant tries to open your pricing page, gets a 403, and builds the comparison from your competitor's page plus whatever third parties say about you. You lose the exchanges where purchase intent is highest.
2. Search-index bots (OAI-SearchBot, PerplexityBot). Cost lands in days to weeks: you drop out of the cited, linked answers these engines produce. This is the closest analogue to deindexing yourself from a search engine — one whose users skew toward exactly the research-heavy queries B2B brands live on.
3. Training bots (GPTBot, ClaudeBot, Google-Extended, CCBot). Cost arrives in months to years, when the next model generation ships knowing your competitors' documentation but not yours. It's invisible until it isn't, and it's the hardest to reverse — you can't retroactively join a training run.

None of these blocks, incidentally, guarantees removal of content already ingested, and robots.txt is a good-faith convention rather than an enforcement mechanism — points worth internalizing before treating a Disallow line as a legal strategy.

The publisher-vs-brand tradeoff

Whether blocking is rational depends entirely on where your revenue comes from.

Publishers sell attention. If AI engines summarize your journalism, the summary substitutes for the visit that funds the newsroom — Pew found users click a traditional result on only ~8% of searches with an AI summary versus ~15% without one.⁶ For a publisher, blocking training bots while negotiating licensing deals is a coherent commercial position, and many large outlets have taken exactly that path.

Brands sell products. Nobody's subscription lapses because ChatGPT accurately described your feature list — that description is the marketing. When an assistant assembles a shortlist of vendors, the winner isn't whoever protected their content; it's whoever the engine knows well enough to recommend. For a brand, an AI engine reading your site is distribution, and blocking it is turning down free syndication of your sales pitch at the exact moment buyers moved their research into chat windows.

The awkward reality: most companies inherit the publisher posture by default — a security-minded engineer pastes a "block all AI" snippet from a blog post, and eighteen months later marketing wonders why competitors dominate every AI-generated comparison. In our experience, that robots.txt archaeology is one of the first things worth checking when a brand's AI visibility is inexplicably low.

A robots.txt for brands that want to be found

Robots.txt is per-user-agent, so you can express a nuanced policy in a few lines. A sensible starting point for a typical B2B brand — welcome answer engines and live browsing everywhere public, keep every bot out of app and account surfaces:

# --- AI answer engines & live browsing: welcome on public pages ---
User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: PerplexityBot
Allow: /

# --- Training crawlers: allowed here; flip to Disallow if you
# --- decide training exposure isn't worth future visibility ---
User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: CCBot
Allow: /

# --- Everyone, including all of the above: stay out of
# --- logged-in surfaces and machine endpoints ---
User-agent: *
Disallow: /app/
Disallow: /account/
Disallow: /api/
Disallow: /cart/

Sitemap: https://www.example.com/sitemap.xml

Implementation notes that save real pain:

• Enumerate, don't wildcard. A blanket User-agent: * Disallow: / mindset ported to AI bots is how brands erase themselves. Name each agent and decide per row of the table above.
• Check the layers above robots.txt. CDN bot rules, WAF settings and one-click blockers override whatever your file says. If Cloudflare's AI-bot toggle is on, your permissive robots.txt is decorative.⁴
• Verify with logs, not vibes. Operators publish IP ranges for their crawlers;¹ confirm the agents you've allowed are actually fetching, and that spoofed traffic isn't polluting your picture.
• Revisit quarterly. New agents appear constantly; a policy written in 2024 silently ignores half of today's list.

The one-click blocking wave, in context

Cloudflare's own data explains why defaults matter so much here. When it launched the one-click block, it reported AI bots hitting roughly 39% of the top million websites it serves — while only about 3% of those sites did anything to block them.⁴ Cloudflare has since pushed further, moving toward blocking AI crawlers by default for new sites and experimenting with pay-per-crawl models.

Read that gap carefully. It means crawler access is increasingly decided not by site owners' intent but by whichever default their CDN shipped last quarter. If AI visibility matters to your pipeline, "we never decided" is now a decision — and probably not the one you'd make on purpose. Meanwhile Gartner's projection of a 25% decline in traditional search volume by 2026 tells you where the forgone visibility would have gone.⁷

A 15-minute decision framework

Run this once with marketing, content and whoever owns the CDN in the room:

1. Classify your revenue. Monetize attention (ads, subscriptions to content) → publisher logic applies; blocking training bots is defensible, and licensing may beat blocking. Monetize products → default to open.
2. Split the tiers. Whatever you decide about training bots, treat search-index and live-browsing agents separately. For brands, blocking OAI-SearchBot or ChatGPT-User is almost never right — that's present-tense invisibility for zero protective benefit.
3. Scope by content, not just by bot. Proprietary research, gated assets and app surfaces can stay disallowed for everyone while product, pricing, docs and comparisons stay open to all.
4. Align the stack. Make robots.txt, CDN bot settings and WAF rules tell the same story; the strictest layer wins. This is the step people skip, because robots.txt is the only layer that's easy to read — the other two only reveal themselves when something actually requests your page with a crawler's user agent. Our free AI access check does exactly that and shows you where the layers disagree.
5. Measure before and after. Baseline your AI mention rate, change policy, re-measure next quarter. Crawler policy is testable — treat it like any other growth lever.

Frequently asked questions