Generate a robots.txt with your sitemap line, the paths you want to keep crawlers out of, and optional rules for AI crawlers.
How to use the robots.txt generator
- Add your sitemap URL and any private paths.
- Choose whether to allow or block AI crawlers.
- Download the file and serve it at /robots.txt.
Keep it short
Most sites only need to block admin, account and API paths and point to the sitemap. Never block CSS or JavaScript files that pages need to render.
What the generated file says
The first group, User-agent: *, applies to every crawler and lists your Disallow paths. If you leave the list empty it says Allow: /, meaning everything may be crawled. Each AI crawler you toggle gets its own group with Disallow: /, which shuts it out of the whole site. Crawl-delay is only added if you set it; Bing and some others respect it, Google ignores it. The last line points to your sitemap at /sitemap.xml under the site URL you entered; edit it if your sitemap lives elsewhere, such as /sitemap_index.xml.
Common robots.txt mistakes
Serving it anywhere but the root: it must be at https://yourdomain.com/robots.txt, and each subdomain needs its own. Blocking a page to hide it from Google: it can still be indexed from links, without a description. Disallowing your whole site on launch because the staging file was copied over. Blocking answer-engine crawlers like OAI-SearchBot or PerplexityBot by accident when you only meant to opt out of training with GPTBot or Google-Extended. Test the result with the robots.txt tester before you deploy.