How to get cited by Claude

To get cited by Claude, the line that matters is the one governing Claude-SearchBot. Anthropic's own support documentation says blocking it may reduce your site's visibility and accuracy in user search results. The crawler most sites actually block is a different one.

In our survey of robots.txt files on 12 September 2026, 15 of 48 sites disallowed ClaudeBot and 9 disallowed Claude-SearchBot.

How to get cited by Claude

To get cited by Claude, allow Claude-SearchBot in robots.txt, decide about ClaudeBot separately since it only governs training, and check whether Brave has indexed you, because Anthropic states that Claude's web search connector calls the Brave Search API. Then put a complete answer in the first paragraph under each heading.

The robots.txt half is three groups rather than two, and the third is a decision most sites make by accident.

# Citations. Anthropic: blocking this may reduce your visibility.
User-agent: Claude-SearchBot
Allow: /

# Training data. Independent of the above.
User-agent: ClaudeBot
Allow: /

# A person pasted your URL into Claude and asked about it.
User-agent: Claude-User
Allow: /

That third group deserves a minute of actual thought rather than being swept in with the others. Disallowing Claude-User means a reader who pastes your link into Claude and asks what it says gets nothing back. For a documentation site that is a reader you are refusing to serve. For a subscription publisher it may be exactly the point.

To see which of the three your own file allows today, put your domain into the AI crawler checker. It reports ClaudeBot, Claude-SearchBot and Claude-User on separate rows, which is what makes a deliberate decision distinguishable from an inherited one.

The second half is the one almost nobody checks, and it is not in your robots.txt at all.

• Claude's web search connector calls the Brave Search API, which Anthropic states in its own help centre. Brave runs its own crawler and its own index of the web.
• So there is a way to be invisible to Claude that no robots.txt change fixes, which is simply not being in Brave's index.
• Brave's webmaster tooling is free and takes about as long to check as Bing's. If you have never looked, that is the gap worth closing first.

One caution, because this is where guides start inventing things. Anthropic does not publish how retrieved results are ranked, weighted, or combined with what the model already knows. Being present in the index Claude queries is a precondition, not a ranking strategy, and anyone selling you the second thing is guessing.

Which Anthropic crawler controls Claude citations?

Claude-SearchBot controls citations. Anthropic describes it as the bot that navigates the web to improve search result quality for users, and warns that disabling it may reduce your site's visibility and accuracy in user search results. ClaudeBot collects training data and Claude-User fetches pages a person asked Claude to visit.

Does Claude use Brave Search?

This is the part missing from every other guide to this question, and it is in Anthropic's own help centre. Claude's web search connector calls the Brave Search API. Anthropic states that the query string, and only the query string, is transmitted: no conversation history, no user identity, no organisation identity, no attached files.

The consequence is worth sitting with. Brave runs its own crawler and its own index of the web; it is not a reskin of Google's results. So there is a route to being invisible in Claude that has nothing to do with your robots.txt at all, which is simply not being in Brave's index.

Nobody should overstate this. Anthropic does not publish how retrieved results are ranked, weighted, or combined with what the model already knows, and the connector is one of several ways Claude can reach a page. But if you have spent two years checking your coverage in Google Search Console and have never once looked at Brave, that is a gap you did not know you had, and it costs nothing to look.

Does Claude respect robots.txt for user-initiated fetches?

Anthropic states that its bots respect do not crawl signals by honoring industry standard directives in robots.txt, and that this covers all three, including Claude-User.

Compare that to the others. OpenAI says robots.txt rules may not apply to ChatGPT-User because a user initiated the request. Perplexity says outright that its Perplexity-User fetcher generally ignores robots.txt for the same reason. Anthropic honours the file even for user-initiated fetches.

If you disallow Claude-User, a person who pastes your URL into Claude will not get your page. That is a real editorial choice, not a technicality.

Which way you should go depends on what your pages are for. A documentation site almost certainly wants Claude-User allowed, because somebody reading your docs through an assistant is a user you are serving. A subscription news site may reasonably decide otherwise.

How many sites block ClaudeBot?

ClaudeBot was the most disallowed crawler in the survey at 15 of 48 sites, 31 percent, against 21 percent for GPTBot. The gap is almost entirely news publishers, several of which disallow Anthropic's training crawler while allowing OpenAI's.

The Guardian is the clearest example of the three-agent decision being made deliberately rather than by accident. Its robots.txt puts anthropic-ai, ClaudeBot, Claude-SearchBot and Claude-User in one disallowed group, and names neither OpenAI crawler anywhere in the file.

Figures come from Aeoly's robots.txt survey of 52 well-known sites, run on 12 September 2026. Each site's robots.txt was read once with our own user agent and all seven crawler tokens were evaluated against that one file. 48 sites returned a readable robots.txt and the other 4 are excluded rather than counted as permissive. The survey reads robots.txt only and never crawls the sites themselves.

The pattern by sector is stark. Every documentation site we could read allows all three Anthropic bots. So does every SaaS marketing site except one, and that one blocks only the training crawlers. The blocking is concentrated in publishers with a licensing position to protect.

How do I check whether Brave has my site?

Search Brave for site: followed by your domain. If the pages you care about are missing from an index Anthropic says the web search connector queries, no robots.txt change will fix it, because the problem is not permission. It is absence.

This takes about a minute and almost nobody has done it, which is the reason it is worth doing. Two years of habit have pointed every check at Google Search Console, and the retrieval layer behind Claude's search is not Google's index.

site:yoursite.com
site:yoursite.com/blog
site:yoursite.com "the phrase your best page answers"

What the three results mean, in order of how bad they are:

• Nothing at all for the domain. You are absent from the index, and that is worth fixing before any writing, because nothing downstream can retrieve what is not there.
• The domain but not the section. Usually a crawl or internal-linking problem rather than a policy one. Check that the section is linked from somewhere Brave can reach and that your sitemap lists it.
• The section but not the page. Ordinary indexing work: the page is thin, orphaned, or too new.

Be careful what you conclude from a good result. Being in Brave's index makes you retrievable; it does not make you cited, and Anthropic publishes nothing about how retrieved results are ranked or combined with what the model already knows. Anyone who tells you they have reverse engineered that has reverse engineered a guess.

How should I structure a page for Claude?

Answer in the first paragraph under each heading, put a name and a date on the work, and state what you did not measure. Claude's own documentation says nothing about content structure, so this is inference from how any retrieval system uses a page, which is a weaker claim than the crawler facts above and worth marking as such.

What is not inference is that the page has to survive being lifted. A passage that only makes sense with the two paragraphs above it attached cannot be quoted without those paragraphs, and nothing quoting you will carry them.

• Lead with the answer. Not the background to the answer, and not a sentence explaining that the question is a good one.
• Put the source inside the sentence. According to Anthropic's crawler documentation travels with a quotation; a hyperlink does not.
• Sign it. A byline and a date give a reader something to weigh, and they cost nothing.
• Say what you do not know. A paragraph naming the limits of your own measurement makes the rest of it more credible rather than less, which is the opposite of how most marketing pages are written.
• Give the figure a method. A number with a stated sample and date can be checked. A number without one is decoration.

That last pair is why this series states its sample size, its date and its four unreadable sites in every post that quotes the survey. It is more useful to a reader and it is harder to argue with.

How do I verify a crawler is really Anthropic?

Anthropic publishes an IP list at claude.com/crawling/bots.json for verification, and notes that blocking by IP may not work correctly or persistently guarantee an opt-out. That caveat is worth reading twice: user agents can be spoofed, and a log line claiming to be ClaudeBot is a claim, not a fact.

We learned this building the survey behind this post. An early version asked each site for robots.txt while sending that crawler's name as its own user agent, and several sites returned a block page instead of the file, because a residential address claiming to be a major crawler is exactly what a spoofed bot looks like. The fix was to stop impersonating crawlers and read the public file honestly.

How do I write pages Claude can quote?

Access decided, the rest is the writing. An assistant needs a passage that survives being lifted out of your page, and a reason to believe it.

• Put a complete answer in the first paragraph under each question heading. Not a preamble, the answer.
• Attribute inside the sentence. According to Anthropic's crawler documentation carries across a quotation in a way a hyperlink does not.
• Put a name on the page. A byline and a stated method give a reader, and anything reading on their behalf, something to check.
• Say what you do not know. A page with an honest limitations paragraph is more quotable than one claiming more than it measured, because the caveat is what makes the rest credible.

How to get cited by Claude: the short version

Allow Claude-SearchBot, decide about ClaudeBot on its own merits, and think properly about Claude-User rather than sweeping it in with the others. Check whether Brave has you. Then write pages whose answers stand on their own. Getting cited by Claude is mostly a question of not having quietly opted out of it.

OpenAI splits the same decision two ways instead of three, which is covered in how to get cited by ChatGPT. If you want the one assistant where nobody is blocking anything, that is how to get cited by Copilot.