Claude-SearchBot
Whether Claude can surface and cite your pages
What Claude-SearchBot does
Claude-SearchBot fetches pages so that Claude can cite them when it searches the web. It is Anthropic's equivalent of OAI-SearchBot, and blocking it removes you from Claude's cited answers.
Anthropic documents all three of its agents in one place and states that they honour robots.txt directives. What it does not publish, unlike OpenAI, Perplexity, Google and Bing, is a list of the IP ranges they crawl from — so verifying a visitor claiming to be Claude-SearchBot is harder than verifying one claiming to be GPTBot.
In our survey, 24 of 148 readable robots.txt files disallowed it, against 42 for ClaudeBot — the same gap, in the same direction, as every other vendor that publishes two tokens.
Claude-SearchBot is not ClaudeBot
The same split OpenAI makes, with names that are harder to tell apart. Claude-SearchBot is citations; ClaudeBot is training.
What ClaudeBot does insteadHow to block Claude-SearchBot
Add this to the robots.txt at the root of your domain, alongside what is already there rather than replacing it.
robots.txt
User-agent: Claude-SearchBot
Disallow: /What that costs you: Claude's search cannot cite you.
To allow it instead, write nothing. A crawler with no group of its own follows your User-agent: * rules, and giving it a named group with Allow: / detaches it from those rules entirely — including any path you disallow there.
Find it in your logs
Anthropic names the token but does not publish a full user-agent string for it, so match on the token itself.
To find every visit it has made, and the addresses it came from:
Terminal
grep -F 'Claude-SearchBot' /var/log/nginx/access.log | awk '{print $1}' | sort | uniq -c | sort -rnCheck that it is really Claude-SearchBot
Harder than it should be. Anthropic documents the token and states that its crawlers honour robots.txt, but publishes no list of the addresses they crawl from — unlike OpenAI, Perplexity, Google and Bing. Reverse DNS and request patterns are what is left. Treat an unverifiable user agent as unverified, because anyone can send any string.
Before you block it by IP
Anthropic warns against this one specifically. Blocking its IP addresses may not work correctly or persistently guarantee an opt-out, because it also stops them reading the robots.txt that carries your opt-out. With this vendor an edge block can defeat the thing it was meant to enforce.
Slow it down instead of shutting it out
If the problem is how hard Claude-SearchBot is hitting you rather than whether it should be here at all, Anthropic supports the non-standard Crawl-delay extension. Most vendors ignore it, so this is worth knowing where it exists: it keeps you visible and takes the load off.
robots.txt
User-agent: Claude-SearchBot
Crawl-delay: 1What Anthropic says about it
It commits to robots.txt
Anthropic states that its bots respect do-not-crawl signals by honouring industry standard directives in robots.txt, and applies that to all three of its agents including the user-triggered one. That is a firmer commitment than OpenAI or Perplexity make about user-initiated fetches.
Read from https://support.anthropic.com/en/articles/8896518 on 14 September 2026.
Should you block Claude-SearchBot?
The same shape as OAI-SearchBot, and the same answer. Blocking it removes you from Claude's cited answers and buys no training protection, because training is ClaudeBot's job.
Anthropic makes the cleanest commitment of any vendor here, stating that all three of its agents honour robots.txt, so a decision you record in that file is likely to be respected.
If load is the problem rather than policy, use Crawl-delay instead. Anthropic supports it, and most vendors do not.
Questions
- How do I verify a visitor is really Claude-SearchBot?
- With more difficulty than for the other vendors, because Anthropic publishes no IP range list. Reverse DNS and rate patterns are what is left. Treat an unverifiable user agent as unverified: anyone can send any string.
The user agent and IP figures on this page were read from Anthropic’s own URLs on 14 September 2026. Blocking figures come from our survey of 170 well-known sites across 10 categories on 14 September 2026, 148 of which returned a readable robots.txt.
Is Claude-SearchBot allowed on your site right now?
The checker reads your live robots.txt and reports every token in this directory, with the exact directive that decided each one.