Fetch a site's live robots.txt and test whether any path is allowed or blocked for a specific crawler, including Googlebot and AI bots.
How to use the robots.txt tester
- Enter the site and the path to test.
- Choose the user agent.
- See which rule matched and whether the path is allowed.
How rules are matched
Crawlers use the group for their own user agent if there is one, otherwise the * group. Within it, the longest matching rule wins, and * and $ wildcards are supported. A single wildcard mistake can block a whole site, so test before you deploy.
Reading the result
The banner says whether your chosen crawler is allowed or blocked on that path, and shows the exact rule that matched and the user-agent group it came from. "No rule matched" means the path is allowed by default. If robots.txt is missing (a 404), everything is allowed. Below that, the same path is tested for Googlebot, Bingbot and the main AI crawlers, such as GPTBot, ClaudeBot, PerplexityBot and Google-Extended, so you can see at a glance who is shut out. The raw file is shown at the bottom. When an allow and a disallow rule are the same length, allow wins, as it does for Google.
Mistakes this catches
Disallow: / left over from a staging site, which blocks everything. A rule without a leading slash. Blocking /*? to hide parameters, which also blocks pages that need them. Forgetting that a crawler with its own group ignores the * group entirely, so a GPTBot section with one rule drops every * rule for GPTBot. Blocking CSS or JS folders that pages need to render. Test the paths you care about most, such as the home page, a product page and the sitemap URL.