AI Crawler Access Checker

Check whether GPTBot, ClaudeBot, PerplexityBot, Google-Extended and other AI bots may crawl your site.

What is an AI crawler?

AI crawlers are bots run by AI companies such as OpenAI, Anthropic, Google and Perplexity. Some collect pages to train models; others fetch pages to answer users' questions in AI search. Sites control their access with robots.txt.

How to use the AI Crawler Access Checker

    Frequently asked questions

    Does blocking Google-Extended remove me from Google Search?

    No. Google-Extended only controls use of your content for Gemini training. Googlebot, which powers Search and AI Overviews, is separate.

    Do AI bots respect robots.txt?

    The major companies say their crawlers follow robots.txt. It is a request, not technical enforcement.

    Should I block AI crawlers?

    It is a business decision. Blocking training bots protects your content; allowing search bots helps you appear in AI answers.