Check my connection

AI & bot directory · AI training

CCBot: what it is, how to verify it and how to block it

CCBot is run by Common Crawl. Builds Common Crawl's free, open archive of the web, which anyone can download, AI developers included.

Key facts

Key facts
Operator
Common Crawl
Category
AI training
robots.txt name
CCBot
Follows robots.txt
OK: Yes
Can it be verified?
OK: Yes, by IP ranges and reverse DNS
Official documentation
Registry checked
Nov 11, 2026

User agent

Its name in robots.txt is CCBot.

Common Crawl documents this user-agent string:

CCBot/2.0 (https://commoncrawl.org/faq/)

Can you verify CCBot?

Yes Common Crawl publishes the IP addresses CCBot uses. We download that list every 6 hours: our copy is from Nov 15, 2026 and has 5 entries; Common Crawl dated it Aug 11, 2026.

Its addresses also have reverse-DNS names under crawl.commoncrawl.org; a name only counts if it points back to the same address.

Check an address from your logs

Does CCBot follow robots.txt?

Yes Common Crawl says CCBot follows robots.txt.

“To prevent Common Crawl from crawling your website, include the following in your robots.txt”

How to block or allow CCBot

To ask it to stay away from the whole site, add this to robots.txt:

User-agent: CCBot
Disallow: /

To keep it out of one folder only:

User-agent: CCBot
Disallow: /private/

To allow it while asking every other bot to stay away:

User-agent: CCBot
Allow: /

User-agent: *
Disallow: /

robots.txt is a request, not a lock. To enforce a block, add a firewall or CDN rule. Because Common Crawl publishes CCBot's addresses, a rule can match them exactly, so a fake CCBot can't slip through as the real one.

To build one robots.txt for several bots at once, use the robots.txt generator.

How often does CCBot visit?

Common Crawl doesn't publish how often CCBot visits a site, and iplover doesn't see your traffic. Your own access log is the only reliable count: paste it into the log checker, which counts each crawler's visits and flags fakes, in your browser.

Official documentation

commoncrawl.org/ccbot

Registry checked Nov 11, 2026. The address lists are downloaded every 6 hours.