AI & bot directory
34 crawlers and AI bots: who runs each one, what it collects, the name to use in robots.txt, whether its operator says it follows robots.txt, and whether you can prove a visit is real. Facts come from each operator's own documentation; where there is none, we say so.
Registry checked Nov 11, 2026. The address lists are downloaded every 6 hours.
Search engine
| Bot | Operator | robots.txt name | What it does | Follows robots.txt | Can it be verified? |
|---|---|---|---|---|---|
| Googlebot | Googlebot |
Crawls the web for Google Search. | Yes | Yes, by IP ranges and reverse DNS | |
| Bingbot | Microsoft | bingbot |
Crawls the web for Bing search. | Not documented | Yes, by IP ranges |
| Applebot | Apple | Applebot |
Crawls the web for Apple's search features in Siri, Spotlight and Safari; the data may also help train Apple's foundation models. | Yes | Yes, by IP ranges and reverse DNS |
| DuckDuckBot | DuckDuckGo | DuckDuckBot |
Crawls the web to improve DuckDuckGo's search results. | Yes | Yes, by IP ranges |
| OAI-SearchBot | OpenAI | OAI-SearchBot |
Finds and links websites in ChatGPT's search answers. It is not used for model training. | Yes | Yes, by IP ranges |
| PerplexityBot | Perplexity | PerplexityBot |
Finds and links websites in Perplexity's search results. Perplexity says it is not used to crawl content for AI foundation models. | Yes | Yes, by IP ranges |
| Claude-SearchBot | Anthropic | Claude-SearchBot |
Navigates the web to improve the quality of search results for Claude's users. | Yes | Yes, by IP ranges |
| Amzn-SearchBot | Amazon | Amzn-SearchBot |
Improves search in Amazon's products and services. Amazon says it does not crawl content for generative AI training. | Yes | Yes, by IP ranges |
| meta-webindexer | Meta | meta-webindexer |
Indexes web pages for search in Meta AI. | Yes | No, user agent only |
AI training
| Bot | Operator | robots.txt name | What it does | Follows robots.txt | Can it be verified? |
|---|---|---|---|---|---|
| Google-Extended | Google-Extended |
A robots.txt control, not a crawler: it decides whether pages Google already crawls may be used to train and ground Gemini models. It does not affect Google Search. | A robots.txt control | Not a crawler | |
| Applebot-Extended | Apple | Applebot-Extended |
A robots.txt control, not a crawler: it decides whether pages Applebot crawls may be used to train Apple's generative models. | A robots.txt control | Not a crawler |
| GPTBot | OpenAI | GPTBot |
Crawls content that may be used to train OpenAI's generative AI models. | Yes | Yes, by IP ranges |
| ClaudeBot | Anthropic | ClaudeBot |
Collects web content that may contribute to training Anthropic's Claude models. It also supports Crawl-delay. | Yes | Yes, by IP ranges |
| CCBot | Common Crawl | CCBot |
Builds Common Crawl's free, open archive of the web, which anyone can download, AI developers included. | Yes | Yes, by IP ranges and reverse DNS |
| meta-externalagent | Meta | meta-externalagent |
Crawls the web for uses such as training Meta's foundation AI models or improving its products by indexing content. | Yes | No, user agent only |
| Amazonbot | Amazon | Amazonbot |
Improves Amazon's products and services; the content may be used to train Amazon's AI models. | Yes | Yes, by IP ranges |
| Bytespider | ByteDance | Bytespider |
ByteDance, TikTok's parent company, has published no documentation for it; community lists name it as a crawler that collects data for AI training. | Not documented | No, user agent only |
| Timpibot | Timpi | Timpibot |
Timpi has published no documentation we could read; community lists describe it as collecting data to train AI models. | Not documented | No, user agent only |
| FacebookBot | Meta | FacebookBot |
An older Meta crawler token that Meta's current crawler documentation no longer lists; community lists still include it for AI training. | Not documented | No, user agent only |
AI assistant fetch
| Bot | Operator | robots.txt name | What it does | Follows robots.txt | Can it be verified? |
|---|---|---|---|---|---|
| ChatGPT-User | OpenAI | ChatGPT-User |
Visits a page when someone asks ChatGPT or a custom GPT about it. It does not crawl the web automatically. | In part | Yes, by IP ranges |
| Perplexity-User | Perplexity | Perplexity-User |
Visits a page when someone asks Perplexity a question that needs it. It is not used for crawling or for training AI models. | Generally no | Yes, by IP ranges |
| Claude-User | Anthropic | Claude-User |
Visits a page when a Claude user asks a question that needs it. | Yes | Yes, by IP ranges |
| meta-externalfetcher | Meta | meta-externalfetcher |
Fetches individual links at a user's request, for example for Meta's AI assistant features. | In part | No, user agent only |
| Amzn-User | Amazon | Amzn-User |
Fetches pages for user actions, such as answering an Alexa question that needs up-to-date information. Amazon says it does not crawl for AI training. | In part | Yes, by IP ranges |
| cohere-ai | Cohere | cohere-ai |
Cohere has published no documentation we could read; community lists describe it as fetching pages to answer users' prompts. | Not documented | No, user agent only |
SEO crawler
| Bot | Operator | robots.txt name | What it does | Follows robots.txt | Can it be verified? |
|---|---|---|---|---|---|
| AhrefsBot | Ahrefs | AhrefsBot, AhrefsSiteAudit |
Crawls the web for Ahrefs' link and SEO database. AhrefsSiteAudit audits sites at their owners' request. | Yes | Yes, by IP ranges and reverse DNS |
| SemrushBot | Semrush | SemrushBot |
Crawls the web for Semrush's SEO and marketing tools. Semrush says it does not use fixed IP blocks. | Yes | No, user agent only |
| DotBot | Moz | dotbot |
Crawls the web for the Moz Link Index behind Link Explorer and the Moz API. | Yes | No, user agent only |
| MJ12bot | Majestic | MJ12bot |
Crawls the web for Majestic's link index. It runs on volunteers' computers, so its addresses are spread worldwide. | Yes | No, user agent only |
Other
| Bot | Operator | robots.txt name | What it does | Follows robots.txt | Can it be verified? |
|---|---|---|---|---|---|
| Google special-case crawlers | – | Google crawlers for specific products, such as checking the landing pages of ads, that work under an agreement with the site. | In part | Yes, by IP ranges and reverse DNS | |
| Google user-triggered fetchers | – | Fetch a page because a person asked a Google product to, for example a feed reader or a site verification. | Generally no | Yes, by IP ranges and reverse DNS | |
| facebookexternalhit | Meta | facebookexternalhit |
Fetches pages shared on Facebook, Instagram or Messenger to show their title, description and image as a preview. | In part | No, user agent only |
| Diffbot | Diffbot | Diffbot |
Diffbot has published no crawler documentation we could read; community lists describe it as extracting web data for its customers. | Not documented | No, user agent only |
| omgili | Webz.io | omgili |
Webz.io's crawler page refused our request, so we couldn't read it; community lists describe it as collecting web data that Webz.io sells. | Not documented | No, user agent only |
How to read this directory
“Follows robots.txt” is what the operator says in its own documentation, not something we measured. Some bots fetch a page because a person asked for it, and their operators say robots.txt may not apply then.
“Can it be verified?” says whether the operator publishes the addresses its bot uses. Where it does, our bot checker can tell a real visit from a fake one. Where it doesn't, the user agent is the only sign, and anyone can copy it.
Bots marked “Not documented” are ones whose operators publish nothing we could read; their user-agent names come from community lists. Every source is on the data sources page.