Robots.txt Generator

Build a valid robots.txt with sitemaps, crawl rules and AI-bot controls.

Runs instantly in your browser — results update as you type.

What is robots.txt?

robots.txt is a plain-text file at the root of a website (example.com/robots.txt) that tells crawlers which paths they may or may not request. It manages crawling — not indexing — and can also point bots to your XML sitemap.

How to use the Robots.txt Generator

    Frequently asked questions

    Where do I put robots.txt?

    In the root of your domain so it loads at https://yourdomain.com/robots.txt. Each subdomain needs its own robots.txt file.

    Does robots.txt stop a page from being indexed?

    No. It stops crawling. A blocked URL can still be indexed from links. Use a noindex meta tag on a crawlable page to keep it out of results.

    Will blocking GPTBot hurt my Google rankings?

    No. GPTBot is OpenAI's crawler. Blocking it has no effect on Google Search, though AI assistants may cite your site less.

    Does Google respect Crawl-delay?

    No. Google ignores Crawl-delay. Bing and Yandex respect it.

    Should I block CSS and JavaScript?

    No. Google needs CSS and JavaScript to render your pages correctly; blocking them can hurt rankings.