The AI access check is a free tool that tells you whether AI crawlers can actually fetch your
site. It reads your robots.txt the way each crawler reads it — a crawler's own rule group
overrides the wildcard group, which is where most of these mistakes hide — then sends real requests using
each crawler's user-agent to catch the blocks robots.txt says nothing about: the CDN rule, the firewall
filter, the JavaScript wall. No signup.
Per-crawler verdicts for GPTBot, ClaudeBot, PerplexityBot, ChatGPT-User, Google-Extended and 19 more
Live user-agent tests that reveal CDN and WAF blocks robots.txt can't show you
An itemised score: every point lost traces to one named cause
3 free checks per hour · results cached 24 hours per site · we identify ourselves on every request.
–
AI access score
Where the points went
Checklist
A one-off check tells you today. Monitoring tells you when it breaks.
The blocks that hurt most are the ones nobody made on purpose — a CDN bot rule enabled during an
incident, a WAF update, a robots.txt line added by a plugin. Sign up free to re-run this on a schedule,
get alerted the moment a crawler's access changes, and see whether the crawlers that can reach you
actually do — matched against the pages engines really cite.
How this check works, exactly. We fetch /robots.txt and evaluate it per crawler
using the standard's group-precedence rules (a crawler's own group overrides the * group), then
request one page with each crawler's user-agent string and compare the response against a normal browser
request. We identify ourselves in every user-agent — we test what your site does with a crawler's
UA, we don't impersonate crawler IP ranges. A crawler we can't get a clear answer for is reported as
unknown, never as allowed. Blocking training crawlers is reported neutrally and
doesn't affect the score: many brands do it deliberately, and it isn't what stops you being cited today.
Common questions
Which crawlers actually matter for being cited?
The ones marked live answer fetch — ChatGPT-User, Claude-User, Perplexity-User. They fire when
somebody asks an assistant a question and it goes to read a page to answer. Blocking those removes you from
answers being written right now. Next are the index crawlers (OAI-SearchBot, PerplexityBot,
Claude-SearchBot, Applebot), which decide whether you can be retrieved at all.
Should I block GPTBot and Google-Extended?
That's a real choice, not a mistake. They collect training data; blocking them keeps your content out of
future model weights without affecting whether you can be cited today. What you should not do is block them
with a rule so broad it catches the live and index crawlers too — which is exactly what this check catches.
My robots.txt allows everything but you found a block. How?
Then something between the crawler and your content is refusing it: a CDN bot-management rule, a WAF, a
rate limiter, or an interstitial challenge page. robots.txt is a request; a firewall is a wall. That gap is
the reason this check sends real requests instead of only reading the file.