Robots.txt Checker

Fetch and validate a robots.txt and test whether a URL is blocked.

What does a robots.txt checker do?

A robots.txt checker downloads a site's robots.txt, finds the rules that apply to a chosen crawler such as Googlebot, and tells you whether a specific URL path is allowed or blocked — and which rule decided it.

How to use the Robots.txt Checker

    Frequently asked questions

    How do I check if a URL is blocked by robots.txt?

    Enter the website, type the path to test (for example /blog/post) and pick a crawler. The checker shows Allowed or Blocked and the rule that decided it.

    What happens if robots.txt does not exist?

    Crawlers assume everything is allowed. A missing file is not an error, but adding one with a Sitemap line is good practice.

    Which rule wins when Allow and Disallow both match?

    The most specific (longest) rule wins. If they are equally long, Allow wins.

    Is robots.txt case-sensitive?

    Paths are case-sensitive, so Disallow: /Admin does not block /admin. User-agent names are matched case-insensitively.

    Why does Google still index a blocked page?

    robots.txt blocks crawling, not indexing. If other sites link to the URL, Google may index it without content. Use noindex on a crawlable page instead.