Skip to content
Doctor SEO
Free SEO audit

Robots.txt Checker: Test Your Rules for Googlebot and AI Crawlers

A free robots.txt checker that reads your file with Google's rules, shows the group and line that decide each URL, and tests Googlebot, Bingbot and 12 AI crawlers at once.

Enter your domain and this robots.txt checker reads the file the way Google’s documentation says Google reads it: the 500 KiB limit, the four fields Google supports, the group each crawler follows, and the rule that decides when two rules match. It points to the exact line behind every finding and tells you how to fix it. Pick a crawler under “Crawl as”, from Googlebot and Bingbot to GPTBot, ClaudeBot and PerplexityBot, and the whole check runs with that crawler’s rules.

We fetch /robots.txt from that host. Enter a full page URL and we also test whether that page can be crawled.

The file is fetched with this crawler's user agent and read with its rules. The access table compares every crawler.

A robots.txt can be perfectly valid and still not do what its author thinks. When we ran the checker on a large US news site on 10 October 2026, its 365-line file had no syntax errors at all. But GPTBot has its own group in that file, so it follows only that group and ignores all 109 rules written for User-agent: *. Applebot has no group of its own and ends up following the Googlebot group. Bingbot follows the * group. None of that is visible from reading the file top to bottom, and it is exactly what the crawler access table shows.

What the robots.txt checker tests

The doctor-seo.net robots.txt checker runs about 80 checks. Each finding links to the document it rests on: Google’s robots.txt documentation, RFC 9309 (the IETF standard for the Robots Exclusion Protocol) or Google’s open-source robots.txt parser. The checks fall into six groups.

Group What it checks Example of an error
Fetch HTTP status for the selected crawler, redirects (up to five), an HTML page served instead of a text file, Content-Type, Cache-Control, size against the 500 KiB limit, UTF-8, byte order mark robots.txt returns 5xx or 429, so Google stops crawling the site
Syntax Lines that aren’t “field: value”, missing colons, misspelled field names, rules before any User-agent line, full URLs or paths without a leading slash in rules, spaces in paths, $ in the middle of a path, repeated rules Disallow: https://example.com/private/, which never matches anything
Groups Which group each crawler follows, groups merged for the same crawler, crawlers that ignore the * rules because they have their own group, groups with no rules, User-agent names that no crawler uses A crawler-specific group that drops every rule in the * group
Rules The whole site blocked for a crawler, Allow and Disallow for the same path, blocked CSS and JavaScript paths, a blocked /wp-admin/admin-ajax.php, unsupported fields such as noindex, crawl-delay and host Disallow: / for Googlebot
Sitemaps Sitemap lines without a full URL, and for the first five listed sitemaps: status for the selected crawler, redirects, whether the file really is a sitemap, other hosts, a Disallow rule that matches the sitemap itself The listed sitemap returns 404
Crawling robots.txt as served to each of 14 crawlers’ user agents, the home page and up to 20 URLs of your choice tested for each crawler, CSS, JavaScript and images on the home page tested against Googlebot’s rules, the http/https and www versions of the host robots.txt refused to one crawler’s user agent while it loads for the others

You can also paste rules, up to 512 KB, to test a robots.txt before you publish it. Add the site URL and the URL tests run against the pasted rules.

How Google reads a robots.txt file

Most robots.txt mistakes come from four rules in Google’s documentation that the file itself doesn’t make obvious. The checker applies all four.

  1. One group per crawler. “Only one group is valid for a particular crawler”: the group with the most specific user agent that matches. A crawler with its own group ignores the * group completely; the two are never combined. If the same crawler has several groups, Google merges them into one.
  2. The longest rule wins. Inside the group, “crawlers use the most specific rule based on the length of the rule path.” Allow: /blog/post beats Disallow: /blog/ for that post because it is longer, wherever it sits in the file.
  3. On a tie, Allow wins. “In case of conflicting rules, including those with wildcards, Google uses the least restrictive rule.”
  4. Only four fields count. Google supports user-agent, allow, disallow and sitemap; “other fields such as crawl-delay aren’t supported.” A noindex line in robots.txt has done nothing in Google since 1 September 2019.

Two details catch people out in practice. Blank lines don’t end a group, so User-agent: Googlebot, an empty line and User-agent: Bingbot form one group that both crawlers share. And a trailing * adds nothing, because every rule already matches everything that starts with its path: /private/* and /private/ are the same rule.

How to read the results

The checker sorts findings into four levels. Errors change how Google crawls the site or block far more than the file’s author probably meant: a 5xx answer, the whole site disallowed for Googlebot, a rule that can never match. Warnings work but cost you something, such as a listed sitemap that doesn’t load or a group that silently drops the * rules. Improvements are tidy-ups, such as redundant wildcards and repeated rules. Good to know is information, such as the Cache-Control header or a Content-Signal line.

Each finding shows the lines it is about, and clicking one jumps to that line in the file viewer, where the group the selected crawler follows is highlighted. The URL test table gives the result for each URL and the line that decides it. “Copy report” copies the whole result as plain text, and the share link reruns the same check with the same crawler.

What happens when robots.txt doesn’t load

The HTTP status of robots.txt matters as much as its content, because Google reads some failures as “crawl everything” and others as “crawl nothing”. The checker requests the file with each crawler’s user agent and reports what each one got.

Answer What Google does
200 Applies the rules.
3xx Follows at least five redirect hops, then treats the file as a 404.
4xx except 429 Treats it “as if a valid robots.txt file didn’t exist”: no crawl restrictions.
5xx, 429 or no answer For the first 12 hours, stops crawling the site and keeps retrying robots.txt. For the next 30 days, uses the last good version.

That is why a firewall that answers 403 to an unrecognised crawler fetching robots.txt doesn’t block it: under Google’s rules a 403 means no restrictions. If the goal is to keep a crawler out, the rule belongs in robots.txt, or the refusal belongs on the pages.

Testing Googlebot, Bingbot and AI crawlers

The access table runs the check for 14 crawlers at once. For each one it shows the group it follows and the line that group starts on, whether the home page and your test URLs are allowed, and what the server answered when robots.txt was requested with that crawler’s user agent.

Group Crawlers in the table What the operator uses them for
Search engines Googlebot, Bingbot, Applebot Web search results
AI search OAI-SearchBot, Claude-SearchBot, PerplexityBot The index behind answers in ChatGPT search, Claude and Perplexity
AI assistants, on user request ChatGPT-User, Claude-User, Perplexity-User Fetching a page a user asked the assistant to read
AI training GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, Meta-ExternalAgent Collecting data for model training

Google-Extended and Applebot-Extended are robots.txt tokens, not crawlers: the rule is honoured, but the fetching is done by Googlebot and Applebot. Some user-triggered fetchers say they may not apply robots.txt to a page a person asked for, and the table notes where that is the operator’s own account.

The checker does not tell you which AI crawlers to allow. Blocking AI training or AI answering bots is a business decision the industry is split on, and doctor-seo.net has not taken a position on it. What the checker shows is whether what you block is what you meant to block.

Every test request comes from the doctor-seo.net server, not from Google’s, Microsoft’s, OpenAI’s or Anthropic’s IP ranges. A firewall that verifies crawlers by IP address can treat the imitation differently from the real crawler. Confirm anything that affects Googlebot with the robots.txt report and URL Inspection in Search Console. In our test of the news site above, the five sitemaps checked answered 403 to the GPTBot user agent from our server; that may be deliberate, or a firewall reacting to an unverified crawler.

Common robots.txt mistakes and how to fix them

  1. A crawler-specific group without the general rules. Adding User-agent: Googlebot with one rule makes Googlebot ignore everything under User-agent: *. Copy the general rules you still want into that group.
  2. Blocking CSS and JavaScript. Google renders pages with their scripts and styles, and its guidance is not to block them. The checker tests the CSS, JavaScript and image files on your home page against Googlebot’s rules.
  3. Using robots.txt to remove pages from Google. A blocked page can still be indexed from links, without its content. To keep a page out of the index, allow crawling and use a noindex meta tag or header.
  4. A Sitemap line with a relative path. The field “must be a fully qualified URL, including the protocol and host.” Write Sitemap: https://example.com/sitemap.xml.
  5. Blocking /wp-admin/admin-ajax.php on WordPress. WordPress’s own default file disallows /wp-admin/ but allows that file, because themes and plugins load content through it.

What this checker cannot tell you

  • Which robots.txt Google has cached. Google generally caches the file for up to 24 hours. The robots.txt report in Search Console shows the version Google fetched and when.
  • Whether a URL is indexed. robots.txt controls crawling, not indexing.
  • What the real crawlers get from their own IP addresses. See the caveat above.
  • Every sitemap in a long list. The first five Sitemap lines are fetched; the XML sitemap checker analyses any of them in full.
  • How crawlers other than Google read edge cases. The rules follow RFC 9309 and Google’s documentation. Other crawlers mostly agree, but not on everything, for example on crawl-delay.

The short version

  • Google reads the first 500 KiB of robots.txt and supports four fields: user-agent, allow, disallow and sitemap.
  • A crawler follows only its most specific group, and a crawler-specific group ignores the * rules.
  • The longest matching rule wins; when two match with the same length, Allow wins.
  • A 4xx answer means no restrictions; a 5xx or 429 stops Google crawling the site.
  • robots.txt blocks crawling, not indexing.
  • Tests from an ordinary server imitate a crawler’s user agent, not its IP address.

FAQ

Is the robots.txt checker free?

Yes, with no account. To keep the server available there is a limit of 15 checks per connection every 10 minutes, and results are cached for 10 minutes; “Check again” fetches a fresh copy.

Why does my rule work here but not in Search Console?

Usually timing. Google may still be using a copy of robots.txt from up to 24 hours ago. The robots.txt report in Search Console shows which version it has and lets you request a recrawl after an urgent change.

Does Google support crawl-delay?

No. Google ignores crawl-delay. Bingbot and some other crawlers read it. To slow Googlebot for a short time, Google documents answering with 500, 503 or 429; our post on Google’s Retry-After documentation covers how.

Are robots.txt paths case-sensitive?

Yes. Paths in Allow and Disallow rules are case-sensitive, so /Private/ does not match /private/. Field names and crawler names are not.

Where should the robots.txt file live?

At the root of each host, for example https://example.com/robots.txt. It applies only to that protocol, host and port, so www.example.com and http://example.com each read their own file. The checker tests both versions and reports whether they redirect or serve a file of their own.

Where to go next

This tool is part of the Doctor SEO tools section, next to the XML sitemap checker. How crawling, robots.txt and indexing fit together belongs to the technical SEO level of the free SEO course, and the SEO glossary defines any term in the report that is new to you.

Sources