Opt out of AI training without losing your citations
The two are separate tokens and almost every "block AI" snippet on the web conflates them. Pick a position, see what it costs, and copy the groups. Nothing is sent anywhere: this runs entirely in your browser.
Start from a position
Or set them one at a time
Decide whether an AI answer can quote you and link back.
OAI-SearchBotOpenAIClaude-SearchBotAnthropicPerplexityBotPerplexityAmzn-SearchBotAmazonDuckAssistBotDuckDuckGoFill the search indexes that Google and Bing, and their AI answers, draw on.
GooglebotGooglebingbotMicrosoftApplebotAppleMistralAI-IndexMistralCollect pages that may be used to train models. Blocking these costs no citations.
GPTBotOpenAIClaudeBotAnthropicGoogle-ExtendedGoogleApplebot-ExtendedAppleAmazonbotAmazonCCBotCommon CrawlFetch a page when a person asks the assistant to open your link.
ChatGPT-UserOpenAIClaude-UserAnthropicPerplexity-UserPerplexityAmzn-UserAmazonMistralAI-UserMistralYour robots.txt
# AI crawlers. Generated by https://aeoly.ai/tools/ai-robots-txt-generator
#
# Paste this into the robots.txt at the root of your domain. Keep your
# existing User-agent: * group: these are additions, not a replacement.
# OpenAI
# Whether your pages can be used to train OpenAI models
User-agent: GPTBot
Disallow: /
# Anthropic
# Whether your pages can be used to train Anthropic models
User-agent: ClaudeBot
Disallow: /
# Google
# Gemini model training and grounding
User-agent: Google-Extended
Disallow: /
# Apple
# Whether your pages train Apple's foundation models
User-agent: Applebot-Extended
Disallow: /
# Amazon
# Improving Amazon's products and services
User-agent: Amazonbot
Disallow: /
# Common Crawl
# The public Common Crawl dataset, which many model builders train on
User-agent: CCBot
Disallow: /
# Allowed by omission, which is how robots.txt works: a crawler with no
# group of its own follows your User-agent: * rules. Naming it here with
# Allow: / would detach it from those rules, so it is left out on purpose.
#
# OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User,
# Googlebot, PerplexityBot, Perplexity-User, bingbot, Applebot,
# Amzn-SearchBot, Amzn-User, DuckAssistBot, MistralAI-Index,
# MistralAI-User
Already have a robots.txt? Check what it permits before you change it. You may already be blocking something you did not mean to.
Check my current robots.txtWhy a generator and not a snippet
Search for how to block AI crawlers and you will find a block of text to paste. It will name eight or ten user agents, it will not say what any of them governs, and it will almost always include OAI-SearchBot beside GPTBot as though they were the same decision.
They are not. One controls whether your writing trains a model. The other controls whether ChatGPT can quote you and send you the reader. Pasting the snippet does the first thing, which you wanted, and the second, which you almost certainly did not.
We measured how often this happens. Of 148 sites whose robots.txt we could read on 14 September 2026, 19 percent blocked GPTBot, which is a defensible choice. 8 percent blocked OAI-SearchBot, which takes them out of ChatGPT's answers altogether. The full survey is here.
So this writes comments into the file next to every group. Not for the crawler, which ignores them, but for whoever opens the file in eight months and has to work out whether the line was deliberate.
Questions
- How do I block AI crawlers in robots.txt?
- Add a User-agent group for each crawler you want to stop, with Disallow: / underneath it. The generator above writes those groups for you. What matters more than the syntax is which tokens you name: blocking GPTBot opts you out of OpenAI's model training, while blocking OAI-SearchBot removes you from ChatGPT's answers. They are different crawlers and most people only mean to do the first.
- How do I opt out of AI training but still get cited?
- Block the training crawlers and leave the retrieval crawlers alone: Disallow GPTBot, ClaudeBot and Google-Extended, and allow OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot and bingbot. That is the second preset above. All three vendors support the split deliberately, which is why they publish separate tokens for it.
- Should I write Allow: / for the crawlers I permit?
- Usually not, and it is not merely redundant. A crawler with no group of its own obeys your User-agent: * rules. The moment you give it a named group it reads that instead and ignores the catch-all entirely, so adding User-agent: GPTBot with Allow: / to a file whose * group disallows /admin quietly grants GPTBot access to /admin. Allowing a crawler is the absence of a rule, not a rule. The generator offers the explicit form for people who want a written record, with that consequence attached.
- Does blocking Googlebot stop AI Overviews?
- Yes, and it stops Google Search along with them, which is why no preset here does it. AI Overviews and AI Mode are built from the ordinary Google index that Googlebot fills. The token that governs Gemini training and grounding is Google-Extended, and blocking that one leaves your rankings and your AI Overview eligibility untouched.
- Where does robots.txt go?
- At the root of the domain, served as plain text: https://example.com/robots.txt. It applies to that host only, so a subdomain needs its own. Paste these groups alongside what is already in the file rather than replacing it, or you will drop the rules the rest of your site depends on.
- Do AI crawlers actually obey robots.txt?
- The documented ones say they do, and OpenAI, Anthropic, Perplexity and Google all publish their tokens so that you can address them. Two caveats worth knowing. Rules addressed to a user-agent nobody publishes do nothing, which is the state of most DeepSeek advice on the web. And a crawler that ignores robots.txt has to be stopped at the network edge instead, because robots.txt is a request rather than a control.
Check the file you have before you replace it
The crawler checker reads your live robots.txt and reports what it currently permits, with the directive that decided each one.