On 17 November 2026 Google published a new page in its crawling documentation: “Allow Google-Agent traffic”. It does not explain how to block Google’s agent. It explains how to make sure the agent gets through.
Google-Agent itself is not new — the user agent has been registered in the crawling changelog since 20 March 2026. The guide is new, and what makes it worth an SEO’s attention is where it says the decision is made. The page is written for the content delivery network, the web application firewall and the bot-detection service. robots.txt does not appear in the body of the document once.
That is not an editorial oversight. It follows from a classification Google already had on the record, and this guide is what makes that classification operational.
What Google actually published
The page has three sections: how Google-Agent identifies its requests, how to allow that traffic in a CDN, firewall or bot-detection service, and a troubleshooting list. It carries the stamp Last updated 2026-11-17 UTC, confirmed on two independent reads on 18 November 2026.
Google defines the agent this way: Google-Agent is the name used by agents hosted on Google infrastructure to navigate the web and perform actions upon user request
. It states the guide’s purpose as follows: Explicitly allowing Google-Agent traffic … helps prevent legitimate requests from being blocked while preserving your existing security controls
.
The crawling changelog logs the publication in a 17 November entry headed Added guidance on allowing Google-Agent
. Its body reads: Published a technical guide on how to ensure that Google-Agent has access to your website
, with the reason given as To help publishers that want to enable Google-Agent for their website
.
The sentence that reframes all of this sits on a different page. The list of Google user-triggered fetchers, stamped Last updated 2026-08-19 UTC, says of the whole family: Because the fetch was requested by a user, these fetchers generally ignore robots.txt rules
. If the agent ignores robots.txt, then allowing or refusing it is a decision executed in the network layer rather than in a text file. The 17 November guide is the manual for that layer.
The four verification rules
Google’s first answer is that some services already do the verification for you: Some bot-detection services, CDNs, and Firewalls provide ways to verify requests made by the Google-Agent user agent
, and it refers site owners to their provider for the exact steps. For anyone without that automatic option, the guide gives four manual rules:
Confirm the User-Agent header contains Google-Agent.
Confirm the IP address as being in user-triggered-agents.json.
If provided, confirm the signed request by verifying the Signature and Signature-Input headers…
, against theSignature-Agentvalue for the identityhttps://agent.bot.goog.For verified requests, skip bot challenges.
The third rule is the newest part. Google signs some of its agent requests using HTTP Message Signatures (RFC 9421) and publishes its key set at https://agent.bot.goog/.well-known/http-message-signatures-directory. The page labels this as unfinished work: Google is also experimenting with the Web Bot Auth protocol using the https://agent.bot.goog identity
.
The order of those rules matters more than it looks. The user-triggered fetchers page carries a Caution: The user agent string can be spoofed
. Allowing on the user agent string alone opens the door to anyone who copies it. The verification that counts is the IP range, or the signature.
We read the range file on 18 November 2026: user-triggered-agents.json declared creationTime: 2026-11-17T15:45:49 and held 20 prefixes — 12 IPv4 and 8 IPv6. PPC Land, reporting a day earlier, described the same file with 20 prefixes split the same way, generated on 16 November at 15:45. Two dated observations 24 hours apart, at the same clock minute, with different generation dates. That is not a measurement of cadence, but it is the reason a hand-copied allowlist ages badly.
The six places a request still dies
The troubleshooting section opens by conceding the real case: If legitimate Google-Agent requests are still blocked after allowing these requests
. Six checks follow, and read together they are a map of where a request falls over in a modern stack:
- Rule priority.
Confirm that your allow rule has higher priority than (and evaluates before) broad cloud or data center ASN blocks
, as well as managed bot rules and category-level AI agent block policies. - Interactive challenges.
Confirm that your rule skips interactive CAPTCHAs and JavaScript challenges in addition to HTTP blocks
, including in-page CAPTCHA widgets triggered on search or form submission. - In-journey actions and APIs.
Confirm that allowing these requests covers actions beyond initial homepage loads
: on-site search, filtering, detail-page clicks, add-to-cart, booking, and the XHR, GraphQL orapi.*endpoints beneath them. - Shared-IP rate limits.
Confirm that per-IP rate-limiting rules account for or exempt verified Google-Agent IP ranges
, because concurrent requests from many users share one egress pool. - IP range updates.
Confirm that IP-based rules stay synced with the latest ranges in user-triggered-agents.json.
- Proxy headers.
Confirm that load balancers and intermediate proxies preserve the original client IP, the User-Agent header
, and, for signed requests, theSignature,Signature-InputandSignature-Agentheaders.
The first item says the most. Google is documenting that category-level AI-agent block policies exist by default inside bot-management products, and that such a policy can outrank a site owner’s own allow rule. That is Google acknowledging that on many sites this decision has already been made by a third party.
What the document does not say
What the page leaves out is as checkable as what it contains. These absences were each confirmed on two independent reads on 18 November 2026.
| Document | Date stamp | What it contributes on Google-Agent |
|---|---|---|
| “Allow Google-Agent traffic” | Last updated 2026-11-17 UTC | Four verification rules and six troubleshooting checks. No robots.txt, no consequences, no named products. |
| List of Google user-triggered fetchers | Last updated 2026-08-19 UTC | What the agent is, its user agent strings, the IP file, and the sentence about ignoring robots.txt. |
| Crawling changelog | Last updated 2026-11-17 UTC | The single entry logging the guide. |
| Search Central documentation changelog | Newest entry 12 November | No 17 November entry and no occurrence of “Google-Agent”. |
| Search Central blog | Newest entry 16 November | Nothing about the guide. |
user-triggered-agents.json |
creationTime 2026-11-17T15:45:49 | 20 prefixes: 12 IPv4 and 8 IPv6. |
Three specific absences, each verified with an explicit yes-or-no question on the second read:
robots.txt is not in the body of the guide. It appears only in the sidebar navigation, as a link to a different page. There is no sentence comparing Google-Agent with Googlebot on that point, and none saying whether the agent obeys the file. The answer lives on the August page, and it is that these fetchers generally do not.
No consequences are documented. The page does not say what a site gains by allowing the agent or loses by blocking it. It does not mention ranking, visibility in AI features, or any effect at all. The closest thing is the purpose sentence quoted above, which describes the benefit of allowing rather than the cost of refusing.
No Google product is named. The guide says agents hosted on Google infrastructure
and stops there. It does not say which features use Google-Agent to act on a user’s behalf.
What it changes in practice
For an SEO manager it changes who you have to ask. For fifteen years the answer to “do we let this bot in?” lived in a file the SEO team could edit. For this family of requests it does not. It lives in the CDN console, in the WAF’s managed rule set, and in the defaults of a bot-detection product — usually owned by a different team, and often by a different company.
It also changes what “we already allowed it” means. The six-item list describes six ways of having allowed the agent and still blocking it: the rule in the wrong place in the evaluation order, covering HTTP blocks only, covering the homepage only, a per-IP limit with no exemption, ranges copied by hand weeks ago, or a proxy that strips the header identifying the agent. None of the six is visible from robots.txt, and none is visible in Search Console.
If you are reviewing crawler rules this month, the robots.txt checker is still the right tool for Googlebot and for the AI crawlers that do obey the file. For Google-Agent it is not, and that distinction is the same one we set out in Claude’s three crawlers and your robots.txt: one vendor can run several agents with different obedience rules, and treating them as a single bot is the commonest mistake.
What to do this week
Seven steps, all runnable with what you already have:
- Grep for the token. Filter your CDN or origin logs for the string
Google-Agentover the last 30 days. Record the request count and the status codes returned. - Read the status codes, not the total. If you see 403s, 429s or challenge responses, you have your answer: the agent is arriving and not getting through.
- Verify before you allow. Do not allow on the user agent string alone — Google warns it can be spoofed. Check the IP against
user-triggered-agents.json. - Automate the range file. Fetch it on a schedule and compare its
creationTime. A hand-copied list can be a day out of date. - Check the evaluation order. Confirm your allow rule is evaluated before data-centre ASN blocks and before any category-level AI-agent policy your provider ships by default.
- Test past the homepage. Repeat the test against on-site search, a filtered listing, a detail page, the cart, and the XHR or GraphQL endpoints behind them. This is the item the guide flags and the one almost nobody checks.
- Write down the decision and its date. Allowing and blocking are both defensible. Not knowing which one is currently in force is not.
This is technical SEO in the literal sense: done at the network layer, verified in the logs.
Where the industry genuinely disagrees
Four real disagreements this document touches and does not resolve. None of them is settled here.
Whether sites should block AI agents at all. One camp holds that an agent acting on a person’s instruction is user traffic, and blocking it means losing that user. The other holds that a site which cannot measure or rate-limit an automated stream should not accept it by default. Nobody has published a revenue comparison between sites that allow and comparable sites that block, so both camps argue from first principles.
Whether a standing instruction is still “user-triggered”. Google’s category is exempt from robots.txt because a person asked for the fetch. What happens when that person asks once and the agent acts for days is a question Google’s documentation does not answer, and whether robots.txt is a usable lever here depends on it.
Whether a documentation change tells you anything about Google’s systems. One camp reads a dated page as the most reliable release note Google publishes. The other answers that Google publishes guidance for site owners rather than specifications, and that the two have diverged repeatedly. Google publishes no mapping from a documentation edit to a serving change, so the rate at which one accompanies the other is unknown.
And a question this document opens. When an agent ignores robots.txt and the decision executes in the CDN, does the publisher still hold that decision, or have they delegated it without noticing to their bot-management vendor’s default? Google’s own guide assumes that category-level AI-agent block policies exist and that your rule must outrank them. That describes a site whose access policy was written by a third party. We have logged it in our contested-topics register, with no verdict.
The author’s opinion
What I will defend here is documentary, and nothing beyond it.
This is the most useful documentation change Google has published this month, and not because of what it teaches you to configure. It is useful because of what it concedes. The six troubleshooting items are not theory; they are the list of things that actually break for anyone who has administered a WAF. That Google had to write them down suggests it hears about them often.
I will not say this signals a ranking change. Google published no measurement, named no feature, promised no outcome and changed no policy, and I am not going to fill that gap with a mechanism I invented.
Nor will I tell you whether to let the agent in. That depends on what a user arriving through an intermediary is worth to your business, and nobody has published the number that would answer it. What I will say is narrower and I think more useful: most sites have not made this decision at all, and as of this week there is a dated document saying exactly where it is being made for them.
What is still unknown
How many sites block Google-Agent today. Google does not publish it. Trade coverage cites security-vendor figures on agent blocking; we did not read those primary sources in this check, so we do not repeat them here.
Which Google features use the agent. The document says “agents hosted on Google infrastructure” and leaves it there.
What blocking costs. No consequence is documented on either page.
When request signing stops being experimental. Google uses that word and gives no date.
And how often the range file changes. We have two dated observations with different creationTime values on consecutive days; that is not a series, and Google publishes no cadence.
Sources
- Google, Allow Google-Agent traffic, crawling documentation, Last updated 2026-11-17 UTC. Read twice on 18 November 2026.
- Google, List of Google user-triggered fetchers, Last updated 2026-08-19 UTC. Read on 18 November 2026.
- Google, crawling infrastructure changelog, entry dated 17 November 2026. Read on 18 November 2026.
- Google, Search Central documentation changelog, newest entry dated 12 November 2026. Read twice on 18 November 2026.
- Google, user-triggered-agents.json, creationTime 2026-11-17T15:45:49. Read on 18 November 2026.
- Barry Schwartz, New Google Search Help Document On How To Allow Google-Agent Traffic, Search Engine Roundtable, 17 November 2026.
- Luis Rijo, Google lists six ways sites still block its AI agent after allowing it, PPC Land, 17 November 2026.
None of these sources is a study, so there is no sample size to cite: they are dated documents and two trade reports. Google published no measurement alongside this guide.