How to use the Robots.txt Generator
Frequently asked questions
Where do I put robots.txt?
In the root of your domain so it loads at https://yourdomain.com/robots.txt. Each subdomain needs its own robots.txt file.
Does robots.txt stop a page from being indexed?
No. It stops crawling. A blocked URL can still be indexed from links. Use a noindex meta tag on a crawlable page to keep it out of results.
Will blocking GPTBot hurt my Google rankings?
No. GPTBot is OpenAI's crawler. Blocking it has no effect on Google Search, though AI assistants may cite your site less.
Does Google respect Crawl-delay?
No. Google ignores Crawl-delay. Bing and Yandex respect it.
Should I block CSS and JavaScript?
No. Google needs CSS and JavaScript to render your pages correctly; blocking them can hurt rankings.

