CCBot
The public Common Crawl dataset, which many model builders train on
CCBot/2.0What CCBot does
CCBot belongs to Common Crawl, a non-profit that publishes a free, openly licensed crawl of the web. It is not a model vendor: it builds a dataset, and model builders take that dataset and train on it.
That makes CCBot the highest-leverage single token in this directory for anyone who wants out of training. One rule refuses a corpus that has fed a great many models, including ones whose own crawlers you have never heard of and cannot name in robots.txt.
It also makes it the bluntest. Common Crawl's archive is used for academic research, web-scale linguistics and search projects as well as commercial model training, and a Disallow does not distinguish between them.
Common Crawl publishes no IP range list, and states on its own page that it is aware of crawlers falsely identifying themselves as CCBot. Traffic claiming the name is not evidence of the name.
How to block CCBot
Add this to the robots.txt at the root of your domain, alongside what is already there rather than replacing it.
robots.txt
User-agent: CCBot
Disallow: /What that costs you: Nothing directly, and a great deal indirectly: this is one refusal that reaches every model trained on the corpus.
To allow it instead, write nothing. A crawler with no group of its own follows your User-agent: * rules, and giving it a named group with Allow: / detaches it from those rules entirely — including any path you disallow there.
Find it in your logs
Common Crawl names the token but does not publish a full user-agent string for it, so match on the token itself.
To find every visit it has made, and the addresses it came from:
Terminal
grep -F 'CCBot' /var/log/nginx/access.log | awk '{print $1}' | sort | uniq -c | sort -rnCheck that it is really CCBot
Harder than it should be. Common Crawl documents the token and states that its crawlers honour robots.txt, but publishes no list of the addresses they crawl from — unlike OpenAI, Perplexity, Google and Bing. Reverse DNS and request patterns are what is left. Treat an unverifiable user agent as unverified, because anyone can send any string.
What Common Crawl says about it
Impersonation is documented
Common Crawl states on its own page that it is aware of crawlers falsely identifying themselves as CCBot. With no published IP list to check against, a CCBot user agent in your logs is a claim rather than a fact.
Read from https://commoncrawl.org/ccbot on 14 September 2026.
Should you block CCBot?
The most effective single line available if your goal is staying out of training data, because it refuses a corpus rather than one company. Nothing about your visibility in any assistant changes.
The argument against is that Common Crawl is also the raw material for academic research and for search projects that are not commercial model training, and one Disallow refuses all of it. Whether that matters is a question about what you object to, rather than about SEO.
Note that it is backward-looking in the way every robots.txt rule is: existing crawls are already published and a rule today does not retract them.
Questions
- Does blocking CCBot stop AI companies training on my site?
- It stops them getting your pages via Common Crawl, which is a substantial route and not the only one. Vendors that run their own training crawlers — GPTBot, ClaudeBot, Google-Extended, Applebot-Extended — need their own rules.
- Does blocking CCBot affect my search rankings?
- No. Common Crawl is not a search engine and its dataset is not an index anyone ranks from.
The user agent and IP figures on this page were read from Common Crawl’s own URLs on 14 September 2026. Blocking figures come from our survey of 170 well-known sites across 10 categories on 14 September 2026, 148 of which returned a readable robots.txt.
Is CCBot allowed on your site right now?
The checker reads your live robots.txt and reports every token in this directory, with the exact directive that decided each one.