Enter your domain
We resolve your URL and fetch robots.txt from the root of your site.
Free AI crawler checker
Are AI crawlers actually allowed to read your site?
We check what your robots.txt says and what your server returns when GPTBot, ClaudeBot, PerplexityBot, Googlebot and CCBot request your pages.
Checked across OpenAI, Anthropic, Perplexity, Google and Common Crawl
OpenAI
Anthropic
Perplexity
GoogleCCCommon Crawl
How it works
Two checks per bot, not just robots.txt
A bot can be allowed in robots.txt but blocked by your CDN, WAF or hosting rules. We test both so you see the full picture.
Parse robots.txt rules
Every User-agent, Allow and Disallow rule is checked for each AI crawler.
Test live bot requests
We request your homepage as each bot and compare the server response to robots.txt.
Coverage
What the AI Crawler Checker does
It checks whether the major AI crawlers are allowed to read your website. For each bot we verify robots.txt policy and the live server response.
10 AI crawlers checked
GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Googlebot, Google-Extended and CCBot.
robots.txt analysis
See exactly which Disallow or Allow rule applies to each bot.
Live server test
We request your page with each bot user-agent and read the HTTP status code.
Mismatch detection
Spot when robots.txt allows a bot but your firewall or CDN blocks it anyway.
FAQ
Questions about the checker
It checks whether major AI crawlers are allowed to read your website. For each bot we check two things: what your robots.txt says, and what your server actually returns when that bot requests a page.
GPTBot, OAI-SearchBot and ChatGPT-User from OpenAI. ClaudeBot, Claude-SearchBot and Claude-User from Anthropic. PerplexityBot. Googlebot and Google-Extended from Google. And CCBot from Common Crawl.
robots.txt can allow a bot while your firewall or CDN blocks it anyway. The live check reveals mismatches between your declared policy and what bots actually experience.
We flag it as Mixed. Usually that means your WAF, bot protection or hosting rules are blocking the user-agent even though robots.txt allows it.
No. This tool is free and runs without signup. For daily AI tracking and competitor benchmarks, book a call or start a BeConsole plan.
Get started
Start with a free snapshot.
Enter your site. Get a prioritized view of your biggest organic opportunities in 48 hours.
- Cross-channel visibility gaps
- Prioritized opportunities vs competitors
- Action plan delivered in 48 hours