How to use the Robots.txt Checker
Frequently asked questions
How do I check if a URL is blocked by robots.txt?
Enter the website, type the path to test (for example /blog/post) and pick a crawler. The checker shows Allowed or Blocked and the rule that decided it.
What happens if robots.txt does not exist?
Crawlers assume everything is allowed. A missing file is not an error, but adding one with a Sitemap line is good practice.
Which rule wins when Allow and Disallow both match?
The most specific (longest) rule wins. If they are equally long, Allow wins.
Is robots.txt case-sensitive?
Paths are case-sensitive, so Disallow: /Admin does not block /admin. User-agent names are matched case-insensitively.
Why does Google still index a blocked page?
robots.txt blocks crawling, not indexing. If other sites link to the URL, Google may index it without content. Use noindex on a crawlable page instead.

