Every AI crawler a vendor actually documents
What each one governs, the user agent it sends, whether there is a published IP list to verify it against, and how many well-known sites block it. Read from the vendors' own URLs on 14 September 2026.
| Crawler | Vendor | Governs | Verifiable | Blocked |
|---|---|---|---|---|
| OAI-SearchBot | OpenAI | Citations | 39 IP prefixes | 12 of 148 |
| GPTBot | OpenAI | Model training | 21 IP prefixes | 28 of 148 |
| ChatGPT-User | OpenAI | User-triggered | 213 IP prefixes | 18 of 148 |
| Claude-SearchBot | Anthropic | Citations | No published list | 24 of 148 |
| ClaudeBot | Anthropic | Model training | No published list | 42 of 148 |
| Claude-User | Anthropic | User-triggered | No published list | 24 of 148 |
| Googlebot | Search index | 317 IP prefixes | 1 of 148 | |
| Google-Extended | Model training | No published list | 32 of 148 | |
| PerplexityBot | Perplexity | Citations | 8 IP prefixes | 33 of 148 |
| Perplexity-User | Perplexity | User-triggered | 4 IP prefixes | 23 of 148 |
| bingbot | Microsoft | Search index | 28 IP prefixes | 1 of 148 |
| Applebot | Apple | Search index | 33 IP prefixes | 3 of 148 |
| Applebot-Extended | Apple | Model training | No published list | 33 of 148 |
| Amzn-SearchBot | Amazon | Citations | Published list | 13 of 148 |
| Amzn-User | Amazon | User-triggered | No published list | 11 of 148 |
| Amazonbot | Amazon | Model training | Published list | 32 of 148 |
| DuckAssistBot | DuckDuckGo | Citations | 486 IP prefixes | 20 of 148 |
| MistralAI-Index | Mistral | Search index | 2 IP prefixes | 10 of 148 |
| MistralAI-User | Mistral | User-triggered | 4 IP prefixes | 20 of 148 |
| CCBot | Common Crawl | Model training | No published list | 44 of 148 |
13 of the 20 publish a list of the addresses they crawl from, which is the only reliable way to tell a real crawler from something wearing its name. Blocking figures come from our survey of 170 well-known sites across 10 categories on 14 September 2026, 148 of which returned a robots.txt we could parse. The full survey, including who blocks what.
Two things most of these lists get wrong
They treat every token as one kind of thing. 5 of the entries here decide whether an assistant can quote you and 4 fill a search index. 6 govern whether your writing trains a model. 5 are fetches made because a person asked for your page. And 2 are not crawlers at all — nothing ever visits your site as Google-Extended or Applebot-Extended, because those are permission tokens read out of your file by a crawler that has another name. A table listing all of this in one column has hidden the only distinction that changes what you should do.
They include tokens nobody publishes. Search for AI crawler lists and you will find DeepSeekBot, usually with a confident robots.txt snippet. DeepSeek publishes no crawler token. A rule addressed to a user agent that does not exist governs nothing, and the file it sits in is now a little harder to read and a little less true.
Every entry here was fetched from a vendor URL on 14 September 2026, and where a vendor publishes an IP list we recorded how many prefixes it held and when they last changed it. Those numbers are there so the claims can be checked, and so we have to check them again when they move.
Which of these can read your site right now?
The checker reads your robots.txt and reports every token in this directory, with the directive that decided each one.