Google documents Retry-After for slowing its crawler: what the page says and what it still does not

Google edited one page of its crawling documentation on 6 October 2026 and gave site owners something they have been improvising for years: an official, worked example of how to tell Googlebot when to come back. The page is Reduce the Google crawl rate, and it now carries the Retry-After HTTP header with two code samples.

It also carries the sentence most of the coverage skipped. Google’s own limit on the technique is one to two days, and the document says what happens past it: “if Googlebot observes these status codes on the same URL for multiple days, the URL may be dropped from Google’s index.”

Four Google documents govern what happens when a server starts refusing Googlebot. Read live on 7 October 2026, exactly one mentions Retry-After and exactly one puts a number on how long the technique is safe. They are the same document, and it is the one that changed yesterday.

What Google’s document actually says

Google’s Reduce the Google crawl rate page, stamped Last updated 2026-10-06 UTC, gives one emergency instruction: “To urgently reduce the crawl rate for a short period of time (for example, a couple of hours, or 1-2 days), return a 500, 503, or 429 HTTP response status code instead of 200 to the crawl requests.”

The effect is not scoped to the URLs that break. The same page states that it “reduces your site’s crawl rate across the whole hostname (for example, subdomain.example.com), including both the URLs that return errors and the URLs that return content. It automatically increases the crawl rate again once the number of these errors is reduced.”

The new material is the header. “When returning a 503 or 429 status code, you can also include a Retry-After HTTP header (as defined in RFC 9110 HTTP Semantics) to indicate when Google’s crawlers can retry the request”, the page says, and it shows both documented forms — a delay in seconds, and an absolute date:

HTTP/1.1 503 Service Unavailable
Retry-After: 120

HTTP/1.1 503 Service Unavailable
Retry-After: Wed, 21 Oct 2026 07:28:00 GMT

Then the limit, verbatim: “We don’t recommend that you do this for a long period of time (meaning, longer than 1-2 days) as it may have a negative effect on how your site appears in Google products.” Search is given as the example, not as the whole scope.

The edit is logged. Google’s crawling documentation changelog carries an entry dated 6 October 2026: “Restructured the emergency crawl rate reduction section and added information and examples about the Retry-After HTTP header to the Reduce the Google crawl rate documentation.”

What the community reported

Barry Schwartz spotted the edit at Search Engine Roundtable on 6 October 2026 and presented it as a documentation change rather than a change in how Googlebot behaves — Google already treated 500, 503 and 429 as crawl-reduction signals before this page mentioned Retry-After. That reading matches the changelog’s own wording, which describes restructuring and adding examples.

No practitioner measurement accompanied the change. Nobody published a before-and-after of crawl volume with and without the header, and Google published no figure for how closely its crawlers honour a Retry-After value. No source in this article is a study, so no sample size applies to any of them: they are documents, and their date stamps are the verification.

Four documents, and what each one is missing

The useful question is not what the new page says but what the other pages still do not. All four were read live on 7 October 2026.

The second row is the gap that matters. How HTTP status codes affect Google’s crawlers is the page a developer reaches first, because it is the page that explains what a 429 means to Google. It says “Google’s crawlers treat the 429 status code as a signal that the server is overloaded, and it’s considered a server error”, and it says “For Google Search, Google’s indexing pipeline removes from the index URLs that persistently return a server error.” It does not define “persistently”, and the string Retry-After does not appear on it at all. Two separate reads of that page on 7 October 2026 returned the same answer to both questions.

So Google’s only published number for how long this is safe sits on a different page from the sentence it qualifies, and the page carrying the consequence has not been touched since 4 February 2026.

The fourth row is the one most SEOs will have missed. The Search Central documentation changelog is the page this industry watches for documentation changes, and its only October 2026 entry is dated 1 October and concerns the generative AI content guide. The crawl-rate edit is logged on a separate changelog, on a separate documentation tree. Anyone monitoring one page was monitoring the wrong one.

What it changes in practice

Nothing about Googlebot’s behaviour changed on 6 October. What changed is that a technique practitioners have used on faith now has a worked example, an RFC reference and a stated expiry attached to it — and that the four available levers turn out to differ in speed by two orders of magnitude.

Returning 5xx or 429 takes effect on the next request. Reporting an unusually high crawl rate through Search Console, which the same Google page points at, carries Google’s own latency: “You cannot request an increase in crawl rate, and it may take several days for the request to be evaluated and fulfilled.”

And robots.txt is not on this ladder at all. The Reduce the Google crawl rate page does not offer it as a rate control, and How Google interprets the robots.txt specification, last updated 2026-08-31 UTC, says Google “generally caches the contents of robots.txt file for up to 24 hours”. A Disallow added while a server is failing may not be read for a day. That is the whole reason the emergency lever is a status code and not a text file, and it is the kind of distinction that separates working technical SEO from folklore. The crawl stage it acts on is the first of the three covered in how Google works.

One scope limit before anyone points this at AI crawlers: this is Google’s documentation about Google’s crawlers. It says nothing about whether any other operator honours a Retry-After value, and controlling other companies’ crawlers is a separate problem with separate tokens, which the course covers in Claude’s three crawlers and your robots.txt.

What to do this week

1. Find out what your origin does under load today. If it returns 200 with a half-rendered page instead of an error, that is the worst available answer: Google keeps crawling at full rate and indexes the damage.
2. If it returns 503 or 429, add the header. Retry-After: 120 for a delay in seconds, or an HTTP date such as Wed, 21 Oct 2026 07:28:00 GMT for an absolute time. Both forms are now in Google’s documentation.
3. Set a timer at 24 hours. Google’s stated ceiling is 1-2 days. The failure mode is an incident that quietly stays open over a weekend, not a deliberate decision to exceed it.
4. Do not scope the errors to a subset of URLs. The reduction applies to the whole hostname either way, and broken URLs are exactly what the index-dropping sentence is about.
5. If the overload is structural rather than a spike, file the Search Console report and plan around several days, per Google’s own wording.
6. Keep robots.txt out of the incident runbook. Up to 24 hours of caching makes it a policy tool, not an emergency brake.

Where the industry genuinely disagrees

Three questions this story touches have no settled answer, and this site does not pretend otherwise.

Does a dated documentation edit tell you anything about Google’s ranking or crawling systems? One camp holds that Google writes documentation to describe what its systems already do, so a dated edit is the closest thing to a release note the industry will ever get. The other holds that guidance and mechanism are different objects that have diverged repeatedly. This case gives the second camp its cleanest exhibit: the changelog entry describes restructuring a section and adding examples, which is language about a document, and nothing in it claims Googlebot behaves differently than it did on 5 October. Nobody can settle it, because Google publishes no mapping from a documentation edit to a serving change.

Is it safe to serve errors to Googlebot deliberately, and for how long? Google’s two relevant documents give a recommendation (“longer than 1-2 days” is not recommended) and an undefined threshold (“persistently”), and those are not the same object. One camp treats a documented technique with a published ceiling as exactly that: usable, within the ceiling. The other treats deliberate error-serving as a risk no site should take while a safer path exists, and points out that the consequence Google names is removal from the index rather than a slower crawl. No published study isolates how many days of 5xx costs how many URLs, so both camps argue from the same absence of data.

Is a figure a Google employee states at a conference evidence about Google’s systems? It matters here because the obvious follow-up question — how long the crawl rate takes to recover — has no documented answer, and the numbers circulating for Google’s discovery and refresh cycles come from a conference recap rather than from a Google document. This site published those figures on 4 October 2026 with their provenance attached and declined to use any of them as a benchmark. That position is unchanged, and what such a figure is worth remains an open question here.

The author’s opinion

What I will defend is the documentary claim and nothing wider. On 7 October 2026, the only Google page carrying both Retry-After and a stated safe duration is the one edited on 6 October, the page that defines what a 429 means to Google has neither, and the changelog most of this industry watches does not mention the edit. That is checkable by anyone from the date stamps, and it cost four fetches.

What I think it means in practice is narrower than the coverage suggests. The edit is a good one — a worked example beats folklore, and an RFC reference beats a forum thread. But I expect to see it read as permission, and the sentence that should change behaviour is not the one about the header. It is the one about multiple days and the index.

I will not tell you that serving errors to Googlebot is safe, because Google’s own two pages do not agree on where the line sits and nobody has measured it. Nor will I claim the edit signals anything about ranking: Google changed a page, and whether anything changed in serving is a question documentation cannot answer.

What is still unknown

• How long recovery takes. The page says the crawl rate “automatically increases” once the errors subside and attaches no duration to that at all.
• What Retry-After actually buys. Google documents the header and publishes no figure for how closely its crawlers honour the value, in either of the two documented forms.
• Where “persistently” sits. That word has governed index removal since 2026-02-04 UTC without a definition, and the 1-2 day figure on the other page is a recommendation about your conduct, not a stated threshold for the indexing pipeline.
• Whether anything outside Search is affected. The warning says “a negative effect on how your site appears in Google products”, with Search named only as an example.

Sources

None of the sources below is a study. They are documents and one trade report, so no sample size applies to any of them; the date stamps are the verification.

• Google, Reduce the Google crawl rate, Crawling infrastructure documentation. Last updated 2026-10-06 UTC. Read live 7 October 2026.
• Google, Crawling documentation changelog, entry dated 6 October 2026. Read live 7 October 2026.
• Google, How HTTP status codes affect Google’s crawlers. Last updated 2026-02-04 UTC. Read twice, live, 7 October 2026.
• Google, How Google interprets the robots.txt specification. Last updated 2026-08-31 UTC. Read live 7 October 2026.
• Google, Search Central documentation changelog. Only October 2026 entry dated 1 October 2026. Read twice, live, 7 October 2026.
• Barry Schwartz, Google Search Crawl Rate Doc Adds Retry-After HTTP & More, Search Engine Roundtable, 6 October 2026.
• Google, Search Status Dashboard, incidents.json read live 7 October 2026: no incident carries a begin timestamp in October 2026.