Check my connection

AI & bot directory

34 crawlers and AI bots: who runs each one, what it collects, the name to use in robots.txt, whether its operator says it follows robots.txt, and whether you can prove a visit is real. Facts come from each operator's own documentation; where there is none, we say so.

Registry checked Nov 11, 2026. The address lists are downloaded every 6 hours.

AI training

BotOperatorrobots.txt nameWhat it doesFollows robots.txtCan it be verified?
Google-Extended Google Google-Extended A robots.txt control, not a crawler: it decides whether pages Google already crawls may be used to train and ground Gemini models. It does not affect Google Search. A robots.txt control Not a crawler
Applebot-Extended Apple Applebot-Extended A robots.txt control, not a crawler: it decides whether pages Applebot crawls may be used to train Apple's generative models. A robots.txt control Not a crawler
GPTBot OpenAI GPTBot Crawls content that may be used to train OpenAI's generative AI models. Yes Yes, by IP ranges
ClaudeBot Anthropic ClaudeBot Collects web content that may contribute to training Anthropic's Claude models. It also supports Crawl-delay. Yes Yes, by IP ranges
CCBot Common Crawl CCBot Builds Common Crawl's free, open archive of the web, which anyone can download, AI developers included. Yes Yes, by IP ranges and reverse DNS
meta-externalagent Meta meta-externalagent Crawls the web for uses such as training Meta's foundation AI models or improving its products by indexing content. Yes No, user agent only
Amazonbot Amazon Amazonbot Improves Amazon's products and services; the content may be used to train Amazon's AI models. Yes Yes, by IP ranges
Bytespider ByteDance Bytespider ByteDance, TikTok's parent company, has published no documentation for it; community lists name it as a crawler that collects data for AI training. Not documented No, user agent only
Timpibot Timpi Timpibot Timpi has published no documentation we could read; community lists describe it as collecting data to train AI models. Not documented No, user agent only
FacebookBot Meta FacebookBot An older Meta crawler token that Meta's current crawler documentation no longer lists; community lists still include it for AI training. Not documented No, user agent only

AI assistant fetch

BotOperatorrobots.txt nameWhat it doesFollows robots.txtCan it be verified?
ChatGPT-User OpenAI ChatGPT-User Visits a page when someone asks ChatGPT or a custom GPT about it. It does not crawl the web automatically. In part Yes, by IP ranges
Perplexity-User Perplexity Perplexity-User Visits a page when someone asks Perplexity a question that needs it. It is not used for crawling or for training AI models. Generally no Yes, by IP ranges
Claude-User Anthropic Claude-User Visits a page when a Claude user asks a question that needs it. Yes Yes, by IP ranges
meta-externalfetcher Meta meta-externalfetcher Fetches individual links at a user's request, for example for Meta's AI assistant features. In part No, user agent only
Amzn-User Amazon Amzn-User Fetches pages for user actions, such as answering an Alexa question that needs up-to-date information. Amazon says it does not crawl for AI training. In part Yes, by IP ranges
cohere-ai Cohere cohere-ai Cohere has published no documentation we could read; community lists describe it as fetching pages to answer users' prompts. Not documented No, user agent only

SEO crawler

BotOperatorrobots.txt nameWhat it doesFollows robots.txtCan it be verified?
AhrefsBot Ahrefs AhrefsBot, AhrefsSiteAudit Crawls the web for Ahrefs' link and SEO database. AhrefsSiteAudit audits sites at their owners' request. Yes Yes, by IP ranges and reverse DNS
SemrushBot Semrush SemrushBot Crawls the web for Semrush's SEO and marketing tools. Semrush says it does not use fixed IP blocks. Yes No, user agent only
DotBot Moz dotbot Crawls the web for the Moz Link Index behind Link Explorer and the Moz API. Yes No, user agent only
MJ12bot Majestic MJ12bot Crawls the web for Majestic's link index. It runs on volunteers' computers, so its addresses are spread worldwide. Yes No, user agent only

Other

BotOperatorrobots.txt nameWhat it doesFollows robots.txtCan it be verified?
Google special-case crawlers Google – Google crawlers for specific products, such as checking the landing pages of ads, that work under an agreement with the site. In part Yes, by IP ranges and reverse DNS
Google user-triggered fetchers Google – Fetch a page because a person asked a Google product to, for example a feed reader or a site verification. Generally no Yes, by IP ranges and reverse DNS
facebookexternalhit Meta facebookexternalhit Fetches pages shared on Facebook, Instagram or Messenger to show their title, description and image as a preview. In part No, user agent only
Diffbot Diffbot Diffbot Diffbot has published no crawler documentation we could read; community lists describe it as extracting web data for its customers. Not documented No, user agent only
omgili Webz.io omgili Webz.io's crawler page refused our request, so we couldn't read it; community lists describe it as collecting web data that Webz.io sells. Not documented No, user agent only

How to read this directory

“Follows robots.txt” is what the operator says in its own documentation, not something we measured. Some bots fetch a page because a person asked for it, and their operators say robots.txt may not apply then.

“Can it be verified?” says whether the operator publishes the addresses its bot uses. Where it does, our bot checker can tell a real visit from a fake one. Where it doesn't, the user agent is the only sign, and anyone can copy it.

Bots marked “Not documented” are ones whose operators publish nothing we could read; their user-agent names come from community lists. Every source is on the data sources page.