ClaudeBot
Whether your pages can be used to train Anthropic models
What ClaudeBot does
ClaudeBot collects pages that may be used to train Anthropic's models. Disallowing it is a training opt-out and does not affect whether Claude can cite you when it searches.
It is the most blocked crawler any vendor runs: 42 of 148 sites disallow it, well ahead of GPTBot at 28. Only CCBot, which belongs to no vendor, is refused more often. Some of that is publishers acting deliberately; some is the name being the most recognisable of Anthropic’s three and therefore the one that gets pasted.
Anthropic states that its crawlers honour robots.txt across all three agents, which is a clearer commitment than some vendors make about user-initiated fetches.
ClaudeBot is not Claude-SearchBot
ClaudeBot governs training only. Blocking it costs no citations; blocking Claude-SearchBot costs all of them.
What Claude-SearchBot does insteadHow to block ClaudeBot
Add this to the robots.txt at the root of your domain, alongside what is already there rather than replacing it.
robots.txt
User-agent: ClaudeBot
Disallow: /What that costs you: Nothing, for citations. Blocking this does not remove you from Claude's search results.
To allow it instead, write nothing. A crawler with no group of its own follows your User-agent: * rules, and giving it a named group with Allow: / detaches it from those rules entirely — including any path you disallow there.
Find it in your logs
Anthropic names the token but does not publish a full user-agent string for it, so match on the token itself.
To find every visit it has made, and the addresses it came from:
Terminal
grep -F 'ClaudeBot' /var/log/nginx/access.log | awk '{print $1}' | sort | uniq -c | sort -rnCheck that it is really ClaudeBot
Harder than it should be. Anthropic documents the token and states that its crawlers honour robots.txt, but publishes no list of the addresses they crawl from — unlike OpenAI, Perplexity, Google and Bing. Reverse DNS and request patterns are what is left. Treat an unverifiable user agent as unverified, because anyone can send any string.
Before you block it by IP
Anthropic warns against this one specifically. Blocking its IP addresses may not work correctly or persistently guarantee an opt-out, because it also stops them reading the robots.txt that carries your opt-out. With this vendor an edge block can defeat the thing it was meant to enforce.
Slow it down instead of shutting it out
If the problem is how hard ClaudeBot is hitting you rather than whether it should be here at all, Anthropic supports the non-standard Crawl-delay extension. Most vendors ignore it, so this is worth knowing where it exists: it keeps you visible and takes the load off.
robots.txt
User-agent: ClaudeBot
Crawl-delay: 1What Anthropic says about it
It commits to robots.txt
Anthropic states that its bots respect do-not-crawl signals by honouring industry standard directives in robots.txt, and applies that to all three of its agents including the user-triggered one. That is a firmer commitment than OpenAI or Perplexity make about user-initiated fetches.
Read from https://support.anthropic.com/en/articles/8896518 on 14 September 2026.
Should you block ClaudeBot?
A real choice, and the most blocked crawler in our survey. Disallowing it is a training opt-out that costs no citations.
Make the decision in robots.txt rather than at the firewall, for the reason above: with this vendor the file is the mechanism, and an IP block can undermine the opt-out it was meant to enforce.
And if the real complaint is crawl volume rather than the training question, slow it down instead. Shutting out a crawler to fix a load problem gives up something you were not trying to give up.
Questions
- Is anthropic-ai the same as ClaudeBot?
- anthropic-ai is an older token that appears in a great many robots.txt files, usually alongside ClaudeBot. Naming it does no harm. ClaudeBot is the token Anthropic documents today, and it is the one that has to be present for a block to do anything.
The user agent and IP figures on this page were read from Anthropic’s own URLs on 14 September 2026. Blocking figures come from our survey of 170 well-known sites across 10 categories on 14 September 2026, 148 of which returned a readable robots.txt.
Is ClaudeBot allowed on your site right now?
The checker reads your live robots.txt and reports every token in this directory, with the exact directive that decided each one.