Googlebot
Google Search, and with it AI Overviews and AI Mode
What Googlebot does
Googlebot is the crawler behind Google Search, and it is on this list because AI Overviews and AI Mode are built from the index it fills. There is no separate AI Overviews crawler to allow or disallow.
That has a blunt consequence: the only way to keep your pages out of AI Overviews through robots.txt is to keep them out of Google Search, because they are the same index. Anybody offering you a token that removes you from one and not the other is describing something that does not exist.
What does exist is nosnippet, max-snippet and data-nosnippet, which limit what Google may quote from a page. Those are snippet controls rather than crawl controls, they apply to ordinary results as well as AI ones, and they are the actual lever for a publisher who wants to be indexed but not extracted.
Google publishes the largest IP range list of any vendor here, 317 prefixes, and also supports reverse DNS verification — a lookup on the visiting address should resolve into googlebot.com or google.com and resolve back again.
Googlebot is not Google-Extended
The most consequential confusion in the category. Googlebot fills the index AI Overviews are built from; Google-Extended governs Gemini training and has no effect on Search or on AI Overviews.
What Google-Extended does insteadHow to block Googlebot
Add this to the robots.txt at the root of your domain, alongside what is already there rather than replacing it.
robots.txt
User-agent: Googlebot
Disallow: /What that costs you: Everything. AI Overviews and AI Mode are built from the Google index, so blocking Googlebot removes you from both as well as from Search.
To allow it instead, write nothing. A crawler with no group of its own follows your User-agent: * rules, and giving it a named group with Allow: / detaches it from those rules entirely — including any path you disallow there.
Find it in your logs
Google names the token but does not publish a full user-agent string for it, so match on the token itself.
To find every visit it has made, and the addresses it came from:
Terminal
grep -F 'Googlebot' /var/log/nginx/access.log | awk '{print $1}' | sort | uniq -c | sort -rnCheck that it is really Googlebot
A user agent is a string the visitor chooses, so on its own it proves nothing — anyone can send yours. Google publishes the addresses Googlebot fetches from: 317 prefixes when we read the file, last changed on 11 September 2026. Take an address out of your logs and check it against that list.
Terminal
curl -s https://developers.google.com/search/apis/ipranges/googlebot.json | python3 -c "
import sys, json, ipaddress
addr = ipaddress.ip_address('203.0.113.10') # the address from your logs
nets = json.load(sys.stdin)['prefixes']
print(any(addr in ipaddress.ip_network(n.get('ipv4Prefix') or n['ipv6Prefix']) for n in nets))
"False means something else is wearing the name, and it is under no obligation to respect anything in your robots.txt. That is the traffic worth blocking at the edge, rather than the crawler that was asking politely.
Should you block Googlebot?
No. Blocking Googlebot removes you from Google Search, and from AI Overviews with it, because they are the same index. Almost nobody who arrives on this page means to do that.
If what you want is to stay indexed but limit what Google may quote, the controls are nosnippet, max-snippet and data-nosnippet. They apply to ordinary results as well as AI ones, and they are the only mechanism that separates being found from being extracted.
If what you want is out of AI Overviews specifically while keeping your rankings, there is no mechanism. Anyone who tells you otherwise is describing a token that does not exist.
Questions
- Can I block AI Overviews without blocking Google Search?
- Not through robots.txt, because they are served from the same index. The nearest thing is nosnippet or max-snippet, which cap what Google may quote while leaving you indexed and ranking.
- Does Google-Extended control whether I appear in AI Overviews?
- No. It governs Gemini training and grounding. This is the most common misunderstanding about Google's AI tokens, and it leads sites to believe they have opted out of something they have not.
The user agent and IP figures on this page were read from Google’s own URLs on 14 September 2026. Blocking figures come from our survey of 170 well-known sites across 10 categories on 14 September 2026, 148 of which returned a readable robots.txt.
Is Googlebot allowed on your site right now?
The checker reads your live robots.txt and reports every token in this directory, with the exact directive that decided each one.