We read the robots.txt of 170 well-known sites
Every documented AI crawler token, checked against every file we could read. The headline is not that publishers are blocking AI, which everybody says. It is which crawler they block most, and it belongs to nobody.
Why we ran this
Almost everything written about AI crawlers is assertion. Publishers are blocking them, we are told, and the web is closing. Very little of it carries a number, and none of it separates the crawler that trains a model from the crawler that quotes you and sends a reader back.
That distinction is the most consequential thing in this corner of the subject. Blocking a training crawler costs you nothing you can see. Blocking a retrieval crawler removes you from an assistant’s answers entirely. The two live in the same vendor documentation, one line apart, and nearly every copy-and-paste snippet on the web treats them as the same decision.
So the question worth measuring was not how many sites block AI but how many block the thing they meant to block. That is answerable, because robots.txt is a public file. It is the rare question in this field you can settle without access to anybody’s analytics.
We also needed the numbers ourselves. They sit on each crawler’s page in our directory, and a directory that says “not measured” on half its entries is a list rather than a reference.
Every token, ranked
The most blocked crawler belongs to nobody
CCBot tops the list at 44 of 148. It belongs to Common Crawl, a non-profit publishing an open dataset that many model builders train on, so one rule refuses a corpus rather than a company and reaches models whose crawlers you have never heard of. The sites doing this have worked out that it is the highest-leverage line in the file.
Nobody blocks a search engine
Googlebot and bingbot are refused by exactly one site each, and it is the same site, which refuses everything. Applebot is refused by 3. Whatever people are trying to do about AI, almost nobody is willing to leave a search engine to do it.
Opting out of training is not losing your citations
Three vendors publish a token for training and a separate one for the crawler that decides whether their assistant can quote you. If sites were pasting snippets blindly, the two figures would track each other. They do not. The training token is refused roughly twice as often in every case.
GPTBottraining, costs nothing visible
OAI-SearchBotcitations, costs the referral
ClaudeBottraining, costs nothing visible
Claude-SearchBotcitations, costs the referral
Amazonbottraining, costs nothing visible
Amzn-SearchBotcitations, costs the referral
That gap is the good news, and it is not the story usually told about this. Most sites that have touched their robots.txt understood the difference and kept the crawler that sends readers back. The interesting number is the residue: the 12 sites that disallow OAI-SearchBot and so cannot be quoted by ChatGPT at all, whatever else they intended.
How to use this if you run a website
Five situations, covering most people who will read this. Find yours.
You have never edited robots.txt for AI crawlers
Then you are in the majority, and by default you are allowing all of them. That is the most citable position available and it requires nothing from you.
Worth confirming rather than assuming, because a CDN, a platform default or a predecessor may have written rules you never saw. That takes about ten seconds with the crawler checker.
You blocked something once and cannot remember what
This is the most common situation and the one that produced the 12 sites above. The usual story is a snippet pasted from a blog post in 2024, naming eight user agents with no explanation of what any of them governs.
Check specifically whether it caught OAI-SearchBot, Claude-SearchBot, PerplexityBot or DuckAssistBot. Those four decide whether an assistant can quote you, and none of them has anything to do with training.
You want out of training but want to keep the citations
This is a supported position at every vendor that publishes two tokens, and it is what the majority in this survey have chosen. Disallow the training set: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, Amazonbot, CCBot.
Start with CCBot if you only do one. It is the open dataset many model builders train from, so refusing it reaches further than refusing any single vendor. Leave the retrieval set alone: PerplexityBot, Claude-SearchBot, DuckAssistBot, Amzn-SearchBot, OAI-SearchBot.
The robots.txt generator writes exactly this as a preset.
You are a publisher weighing whether AI answers replace your traffic
Then you are in the one category where blocking retrieval is a mainstream choice rather than an accident. News sites block at least one crawler at 85 percent, and the named lists further down are mostly newsrooms.
The honest version of this decision is about substitution, not about training or privacy. If an assistant can answer the question your page answers without sending anybody to you, the citation is worth less than the click it replaced. That is a judgement about your format, and recipes, symptom pages and reference material sit at the exposed end of it. Note that travel and lifestyle is already at 70 percent and health at 56 percent, which suggests those sectors reached the same conclusion before anybody wrote about it.
You run documentation, a SaaS product or anything people look things up in
Your category blocks essentially nothing. Documentation is at zero of 19, SaaS at 2 of 29, education at 1 of 13.
If your robots.txt does block an AI crawler, you are an outlier in your own category and it is worth finding out whether that was a decision or an inheritance. Nobody writing developer documentation has ever wanted fewer people to find the answer.
What to actually do, in order
- 1
Find out what your file says now
Not what you think it says. Read the live file, and check every token rather than the two you remember. A block you inherited is still a block.
Check my robots.txt - 2
Decide training and citations separately
They are different questions with different costs and different tokens. Answering them together is the single mistake this survey was built to measure, and it always resolves the same way: you keep the training opt-out you wanted and lose the citations you did not know you had.
- 3
Leave the search crawlers alone
Googlebot, bingbot and Applebot are search crawlers before they are AI ones. AI Overviews come from the Google index, so there is no way to leave the feature without leaving the search engine. One site in this survey blocks them, and it blocks everything.
- 4
Write the file, with comments
Name what each group is for, in the file, next to the group. Not for the crawler, which ignores comments, but for whoever opens it in eight months and has to work out whether the line was deliberate.
Generate the groups - 5
Wait, then check again
OpenAI states changes take about 24 hours to reach its systems. DuckDuckGo states 72. A block that appears not to have worked on day one has usually just not propagated, and changing something else in the meantime makes the result impossible to read.
Who shuts out the crawlers that would cite them
Named, because a percentage is an assertion and a list of domains is evidence. These sites disallow a retrieval crawler at the root, which is the token deciding whether that assistant may quote and link them.
aljazeera.com, allrecipes.com, amazon.com, bbc.co.uk, bloomberg.com, bonappetit.com, cnn.com, drugs.com, ebay.com, forbes.com, ft.com, imdb.com, investopedia.com, nature.com, newyorker.com, nytimes.com, pinterest.com, politico.com, quora.com, reddit.com, reuters.com, scientificamerican.com, seriouseats.com, telegraph.co.uk, theatlantic.com, theguardian.com, theverge.com, verywellhealth.com, vox.com, washingtonpost.com, wired.com, wsj.com, yelp.com
allrecipes.com, amazon.com, bloomberg.com, bonappetit.com, cnn.com, imdb.com, investopedia.com, newyorker.com, nytimes.com, pinterest.com, politico.com, quora.com, reddit.com, reuters.com, seriouseats.com, telegraph.co.uk, theatlantic.com, theguardian.com, theverge.com, verywellhealth.com, vox.com, wired.com, wsj.com, yelp.com
allrecipes.com, amazon.com, bloomberg.com, bonappetit.com, cnn.com, investopedia.com, newyorker.com, nytimes.com, politico.com, reddit.com, reuters.com, seriouseats.com, telegraph.co.uk, theguardian.com, theverge.com, verywellhealth.com, vox.com, wired.com, wsj.com, yelp.com
allrecipes.com, bbc.co.uk, cnn.com, healthline.com, investopedia.com, pinterest.com, reddit.com, reuters.com, seriouseats.com, theguardian.com, verywellhealth.com, wsj.com, yelp.com
amazon.com, bbc.co.uk, bloomberg.com, cnn.com, imdb.com, nytimes.com, pinterest.com, politico.com, quora.com, reddit.com, telegraph.co.uk, yelp.com
News and recipes dominate both lists, which fits. They are the two kinds of page an assistant can most easily answer without sending anybody anywhere. Whether that makes blocking the right call is a judgement about substitution, and not one this survey can make for you.
It is almost entirely a publishing decision
Share of readable sites in each category disallowing at least one of the 20 tokens.
Documentation is at zero of 19, the cleanest result in the survey. Nobody writing developer documentation has ever wanted fewer people to find the answer, and it shows.
The two figures we did not expect are travel and lifestyle at 70 percent and health at 56 percent. Recipe pages and symptom pages are the two formats an assistant can answer completely, so the visit never happens. Those categories appear to have reached that conclusion well before anybody wrote about them, and they are the part of this survey we would most like somebody else to replicate.
How this was measured
Figures come from Aeoly's robots.txt survey of 170 well-known sites across 10 categories, run on 14 September 2026. Each site's robots.txt was read once with our own user agent and all twenty documented crawler tokens were evaluated against that one file. 148 sites returned a readable robots.txt and the other 22 are excluded rather than counted as permissive. The survey reads robots.txt only and never crawls the sites themselves.
The method matters more than usual, because the obvious way to run this survey is wrong. An earlier version of ours sent each crawler’s own token as its User-Agent, reasoning that this is what the real crawler does. Ask a large site for robots.txt while calling yourself bingbot from an address bingbot does not use, and you get a block page, because that is exactly what a spoofed crawler looks like. The parser then found no rules in that HTML and recorded the site as allowing everything, for precisely the sites blocking hardest.
So this reads each file once, honestly, as AeolyBot, and evaluates every token against those same bytes. That is also how robots.txt works: one public document whose User-agent groups select which rules apply.
The 22 sites we could not read, and why
adidas.com HTTP 403apnews.com HTTP 403argos.co.uk HTTP 403asos.com TimeoutErrorcostco.com TimeoutErrordictionary.com HTTP 403economist.com HTTP 403indiehackers.com HTTP 403khanacademy.org not a robots.txtlinkedin.com not a robots.txtlowes.com HTTP 403macys.com HTTP 403mayoclinic.org HTTP 403moneysavingexpert.com HTTP 403npr.org TimeoutErrorox.ac.uk HTTP 403schwab.com not a robots.txtsephora.com HTTP 403stackexchange.com HTTP 418stackoverflow.com HTTP 418webflow.com TypeErrorzara.com HTTP 403Most of these are bot management refusing an unfamiliar user agent, which is itself a finding of a kind: several large sites will not serve their own robots.txt to anything they do not recognise. They are excluded from every denominator rather than guessed at.
What this report cannot tell you
Whether a crawler is actually blocked
robots.txt is one gate. A site can refuse a crawler at the CDN, by IP range or with a bot challenge, and write nothing in the file. Several of the sites we could not read are probably doing that to us. This measures stated policy, not enforcement.
Whether crawlers obey it
The documented ones say they do, and Perplexity says openly that its user-triggered fetcher generally does not. A Disallow is a request. Where you need a control, the enforcement point is your network edge.
Whether allowing a crawler gets you cited
Access is a precondition, not a cause. A crawler that cannot read you certainly cannot quote you. One that can will quote you only if the page answers something clearly enough to be worth quoting.
What the whole web does
This is 170 well-known sites chosen by us across ten categories, not a random sample. It describes the sites large enough to have had this argument internally. A long-tail sample would almost certainly block far less, because most sites have never touched the file at all.
Which way things are moving
This is one run on one date. We have an earlier and narrower run, and we deliberately do not compare them, because they sampled different site lists. Two measurements of two different things is not a trend.
Why any given site decided this
A robots.txt records what somebody chose, never why. Some of these entries are the outcome of a licensing negotiation, some are a considered policy, and some are a snippet pasted by a contractor in 2024. The file looks identical in all three cases.
Use these numbers
The data is free to quote and free to argue with. If you use a figure, this is the attribution we would like, and the link is what lets your readers check it.
Attribution
Aeoly, "AI crawler blocking across 170 websites" (14 September 2026).
https://aeoly.ai/research/ai-crawler-blockingEvery per-crawler figure also appears on that crawler’s own page in the directory, alongside its user agent, published IP ranges and what blocking it costs. The survey script is in the repository, so the run can be repeated and disagreed with. If you replicate it and get something different, we would rather hear about it than not.
Start with your own file
The checker reads it and reports all 20 of these tokens, with the exact directive that decided each one. No account, and it takes about ten seconds.