AI Crawler Access Checker
Enter your site and up to two key pages. We read your robots.txt and page tags. You see which search and AI crawlers can reach each page, and the rule that blocks the rest.
Run the check- robots.txt and page tags
- Search and AI crawlers
- The exact blocking rule
Which crawlers the checker tests
Search crawlers decide whether you appear in answers: Googlebot, OAI-SearchBot (ChatGPT search), Claude-SearchBot and PerplexityBot. Block them and you drop out of those answers.
Training crawlers collect content for future models: GPTBot and ClaudeBot. You can block them as a business choice, and you still show up in search answers.
Google-Extended is a robots.txt token with no crawler of its own. It controls whether Google may use your content for Gemini, and has no effect on Google Search.
What the checker reads and can't see
It reads robots.txt, the meta robots tags in each page's head and the X-Robots-Tag header. It can't see firewall or CDN rules that block some bots by IP or user agent. Those rules can still block a page this checker marks allowed, so check your CDN's bot settings too.
Common questions
How do I know if GPTBot can crawl my site?
Enter your site. The checker applies your robots.txt rules to GPTBot and the other crawlers for each page. It shows allowed or blocked, with the rule that decides it.
Should I block AI crawlers?
Blocking search crawlers such as OAI-SearchBot or PerplexityBot keeps you out of their answers. Blocking training crawlers such as GPTBot only stops future model training. Most brands that want AI visibility allow the search crawlers.
What is Google-Extended?
A robots.txt token that controls whether Google may use your content for Gemini models. It doesn't change crawling for Google Search or AI Overviews.
Why does a page show as unknown?
The checker couldn't read your robots.txt, for example because the server returned an error or a bot challenge page. Fix that first, since crawlers see the same response.
Does allowing crawlers get me cited by AI?
No, it only makes a citation possible. The page still needs clear answers, facts the model can trust and links from other sites.
Do you store the sites I check?
No. We keep a result for 10 minutes so a repeat check loads at once, then we delete it. We fetch only public pages, never private addresses.
Free AI visibility review
Find what blocks your key pages
Share your site. We check crawler access, page tags and what AI answers cite instead of you, and reply within 24 hours.
- robots.txt and CDN bot rules
- Pages AI answers can read
- Where competitors get cited
- The first pages to fix
Cancel with 30 days’ written notice. You keep every account and all your data.
You're All Set!
We have your request.
Expect a reply from our team within 24 hours. Want to talk sooner? Pick a time below.