Check my connection

Guides · Investigate an IP

How to block AI crawlers (GPTBot, ClaudeBot, PerplexityBot and more): IP ranges and robots.txt

Your server logs fill up with GPTBot, ClaudeBot and meta-externalagent, and you want a say in what happens to your content. This page covers the major AI crawlers, the ones in our bot registry, as each operator documents them, with copy-ready rules for robots.txt, nginx, Apache and Cloudflare. New bots appear all the time, so no list is complete; the tables grow with the registry, the same data behind Verify a bot and the robots.txt generator.

Two dates on this page: "Last updated" at the top is when this article was last revised. "Registry checked", under each table, is when we last checked every bot in the registry against its operator's documentation. The official IP lists behind the tables are downloaded again every 6 hours.

Three kinds of AI bot, three different decisions

"Blocking AI" can mean three things, and each one is a different bot with its own robots.txt token:

Decide what you want first

What to block for each goal
If you want to…BlockKeep allowed
Keep your content out of AI training, but stay in AI search answersThe training groupThe AI search crawlers, Googlebot, Applebot
Stay out of AI answers and citations as wellThe training and AI search groupsGooglebot, Bingbot, Applebot
Stop assistants from reading a page when a user asksThe user-triggered group, at the server or firewallEverything else
Keep your Google and Apple search rankingsNever Googlebot or Applebot: use Google-Extended and Applebot-Extended insteadGooglebot, Applebot

The AI crawlers, bot by bot

For each bot: the operator, the token to use in robots.txt, whether the operator says it follows robots.txt, whether a request can be proven to come from it, the official IP list where there is one, and the operator's documentation. Each name links to the bot's page in our directory, with its full user agent strings and verification details.

Training crawlers

AI training crawlers and AI-training controls. Registry checked November 11, 2026. The address lists are downloaded every 6 hours.
BotOperatorrobots.txt nameFollows robots.txtCan it be verified?Sources
Google-Extended Google
Google-Extended
A robots.txt control Not a crawler
Operator's documentation: developers.google.com
Applebot-Extended Apple
Applebot-Extended
A robots.txt control Not a crawler
Operator's documentation: support.apple.com
GPTBot OpenAI
GPTBot
Yes Yes, by IP ranges
Official IP list: openai.com/gptbot.jsonOperator's documentation: developers.openai.com
ClaudeBot Anthropic
ClaudeBot
Yes Yes, by IP ranges
Official IP list: claude.com/crawling/bots.jsonOperator's documentation: support.claude.com
CCBot Common Crawl
CCBot
Yes Yes, by IP ranges and reverse DNS
Official IP list: index.commoncrawl.org/ccbot.jsonOperator's documentation: commoncrawl.org
meta-externalagent Meta
meta-externalagent
Yes No, user agent only
Operator's documentation: developers.facebook.com
Amazonbot Amazon
Amazonbot
Yes Yes, by IP ranges
Official IP list: developer.amazon.com/amazonbot/ip-addresses/Operator's documentation: developer.amazon.com
Bytespider ByteDance
Bytespider
Not documented No, user agent only
–
Timpibot Timpi
Timpibot
Not documented No, user agent only
–
FacebookBot Meta
FacebookBot
Not documented No, user agent only
–

Two rows are tokens, not crawlers. Google says Google-Extended doesn't have a separate HTTP request user agent string. (source), and Apple that Applebot-Extended does not crawl webpages. (source) Googlebot and Applebot do the fetching; the token only decides whether the result may be used for AI. Apple adds that Webpages that disallow Applebot-Extended can still be included in search results. (source)

Bytespider, Timpibot and FacebookBot have no public documentation we could read, so there is no robots.txt promise to rely on and no IP list to check against. If they matter to you, block them at the server.

AI search crawlers

AI search crawlers. Registry checked November 11, 2026. The address lists are downloaded every 6 hours.
BotOperatorrobots.txt nameFollows robots.txtCan it be verified?Sources
OAI-SearchBot OpenAI
OAI-SearchBot
Yes Yes, by IP ranges
Official IP list: openai.com/searchbot.jsonOperator's documentation: developers.openai.com
PerplexityBot Perplexity
PerplexityBot
Yes Yes, by IP ranges
Official IP list: perplexity.ai/perplexitybot.jsonOperator's documentation: docs.perplexity.ai
Claude-SearchBot Anthropic
Claude-SearchBot
Yes Yes, by IP ranges
Official IP list: claude.com/crawling/bots.jsonOperator's documentation: support.claude.com
Amzn-SearchBot Amazon
Amzn-SearchBot
Yes Yes, by IP ranges
meta-webindexer Meta
meta-webindexer
Yes No, user agent only
Operator's documentation: developers.facebook.com

OpenAI is explicit that the two opt-outs are separate: a webmaster can allow OAI-SearchBot in order to appear in search results while disallowing GPTBot (source). It also notes that when both are allowed, we may use the results from just one crawl for both use cases (source).

User-triggered fetchers

Fetchers acting on a user's request. Registry checked November 11, 2026. The address lists are downloaded every 6 hours.
BotOperatorrobots.txt nameFollows robots.txtCan it be verified?Sources
ChatGPT-User OpenAI
ChatGPT-User
In part Yes, by IP ranges
Official IP list: openai.com/chatgpt-user.jsonOperator's documentation: developers.openai.com
Perplexity-User Perplexity
Perplexity-User
Generally no Yes, by IP ranges
Official IP list: perplexity.ai/perplexity-user.jsonOperator's documentation: docs.perplexity.ai
Claude-User Anthropic
Claude-User
Yes Yes, by IP ranges
Official IP list: claude.com/crawling/bots.jsonOperator's documentation: support.claude.com
meta-externalfetcher Meta
meta-externalfetcher
In part No, user agent only
Operator's documentation: developers.facebook.com
Amzn-User Amazon
Amzn-User
In part Yes, by IP ranges
cohere-ai Cohere
cohere-ai
Not documented No, user agent only
–

Read the "follows robots.txt" column carefully here. OpenAI writes that for ChatGPT-User Because these actions are initiated by a user, robots.txt rules may not apply. (source) Perplexity says its user fetcher generally ignores robots.txt rules (source), and Meta that its fetcher may bypass robots.txt rules (source). If you want these out, robots.txt alone won't do it.

Ready-to-copy robots.txt

robots.txt lives at the root of each host: https://example.com/robots.txt covers example.com only, so www, shop and every other subdomain need their own copy. Several User-agent lines may share one rule, as RFC 9309 allows. Copy the groups that match your decision.

Training crawlers and the AI-training controls:

# AI training crawlers and AI-training controls
# From the iplover.com/bots registry, checked 2026-11-11
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: meta-externalagent
User-agent: Amazonbot
User-agent: Bytespider
User-agent: Timpibot
User-agent: FacebookBot
Disallow: /

AI search crawlers (only if you also want to leave AI answers):

# AI search crawlers: blocking these removes you from their answers
# From the iplover.com/bots registry, checked 2026-11-11
User-agent: OAI-SearchBot
User-agent: PerplexityBot
User-agent: Claude-SearchBot
User-agent: Amzn-SearchBot
User-agent: meta-webindexer
Disallow: /

User-triggered fetchers (a polite request; see above for why it may be ignored):

# Fetchers acting on a user's request (some may not follow robots.txt)
# From the iplover.com/bots registry, checked 2026-11-11
User-agent: ChatGPT-User
User-agent: Perplexity-User
User-agent: Claude-User
User-agent: meta-externalfetcher
User-agent: Amzn-User
User-agent: cohere-ai
Disallow: /

Each block disallows the whole site. To keep a section open, add an Allow: line for its path in the same group. Under RFC 9309 the most specific (longest) matching rule wins, so Allow: /blog/ beats Disallow: / for pages under /blog/.

robots.txt is a request, not a lock

robots.txt works because well-behaved crawlers choose to read and follow it. It doesn't stop a scraper that ignores it, and a scraper that sends a browser's user agent never matches your User-agent lines in the first place. Treat robots.txt as the statement of your wishes that documented crawlers honour, and enforce it where it matters.

Enforce it at the server or firewall

A user-agent rule refuses every request whose user agent contains one of the tokens. It catches honest bots on their next request, and lazy fakes too. The rules below use the same three groups as the robots.txt blocks above, so they enforce the same choice:

Every rule keeps /robots.txt readable, so crawlers can still see your opt-out.

nginx

The first map sets $ai_bot to 1 when the user agent contains a token of an active group (~* means a case-insensitive match). The second map passes that value on for every path except /robots.txt. The if then returns 403 to those requests. To switch on another group, remove the # in front of its pattern line.

# In the http { } block.
# 1. Which requests come from a chosen bot (case-insensitive match)
map $http_user_agent $ai_bot {
    default 0;
    # Training crawlers (the first robots.txt block)
    "~*(GPTBot|ClaudeBot|CCBot|meta-externalagent|Amazonbot|Bytespider|Timpibot|FacebookBot)" 1;
    # User-triggered fetchers: uncomment to refuse them too
    # "~*(ChatGPT-User|Perplexity-User|Claude-User|meta-externalfetcher|Amzn-User|cohere-ai)" 1;
    # AI search crawlers: uncomment only if you also leave AI answers
    # "~*(OAI-SearchBot|PerplexityBot|Claude-SearchBot|Amzn-SearchBot|meta-webindexer)" 1;
}
# 2. Refuse them everywhere except /robots.txt, so they can read your opt-out
map $uri $block_ai {
    default $ai_bot;
    /robots.txt 0;
}

# In each server { } block that should refuse them:
if ($block_ai) { return 403; }

Apache

Each SetEnvIfNoCase line marks requests from one group; uncomment a line to add its group. The SetEnvIf line removes the mark for /robots.txt, and Require not env refuses the marked requests with 403. In .htaccess this needs AllowOverride FileInfo AuthConfig (or All).

# <VirtualHost>, <Directory> or .htaccess (Apache 2.4; mod_setenvif, mod_authz_core)
# Training crawlers (the first robots.txt block)
SetEnvIfNoCase User-Agent "(GPTBot|ClaudeBot|CCBot|meta-externalagent|Amazonbot|Bytespider|Timpibot|FacebookBot)" block_ai
# User-triggered fetchers: uncomment to refuse them too
# SetEnvIfNoCase User-Agent "(ChatGPT-User|Perplexity-User|Claude-User|meta-externalfetcher|Amzn-User|cohere-ai)" block_ai
# AI search crawlers: uncomment only if you also leave AI answers
# SetEnvIfNoCase User-Agent "(OAI-SearchBot|PerplexityBot|Claude-SearchBot|Amzn-SearchBot|meta-webindexer)" block_ai
# Keep /robots.txt readable, so they can read your opt-out
SetEnvIf Request_URI "^/robots\.txt$" !block_ai
<RequireAll>
    Require all granted
    Require not env block_ai
</RequireAll>

Cloudflare

Create one WAF custom rule per group you blocked in robots.txt, with the action Block. Each expression matches that group's tokens, plus the bots Cloudflare has verified itself in the matching verified bot category, and leaves /robots.txt alone.

Rule 1, training crawlers (the first robots.txt block). Cloudflare category: AI Crawler.

(http.user_agent contains "GPTBot" or http.user_agent contains "ClaudeBot" or http.user_agent contains "CCBot" or http.user_agent contains "meta-externalagent" or http.user_agent contains "Amazonbot" or http.user_agent contains "Bytespider" or http.user_agent contains "Timpibot" or http.user_agent contains "FacebookBot"
 or cf.verified_bot_category eq "AI Crawler")
and http.request.uri.path ne "/robots.txt"

Rule 2, optional, user-triggered fetchers. Cloudflare category: AI Assistant.

(http.user_agent contains "ChatGPT-User" or http.user_agent contains "Perplexity-User" or http.user_agent contains "Claude-User" or http.user_agent contains "meta-externalfetcher" or http.user_agent contains "Amzn-User" or http.user_agent contains "cohere-ai"
 or cf.verified_bot_category eq "AI Assistant")
and http.request.uri.path ne "/robots.txt"

Rule 3, optional, AI search crawlers: only if you also want to leave AI answers. Cloudflare category: AI Search, which Cloudflare keeps for existing rules but no longer separates from ordinary search in its newer classification, so the tokens do most of the work here.

(http.user_agent contains "OAI-SearchBot" or http.user_agent contains "PerplexityBot" or http.user_agent contains "Claude-SearchBot" or http.user_agent contains "Amzn-SearchBot" or http.user_agent contains "meta-webindexer"
 or cf.verified_bot_category eq "AI Search")
and http.request.uri.path ne "/robots.txt"

Cloudflare decides which verified bots belong to each category, and a category can include bots that aren't in our tables. If you want a rule to match only the listed tokens, delete its cf.verified_bot_category line.

Cloudflare also has a dashboard setting for this, now called AI Crawler and Agent access policies (source), under Security, Settings. It sorts bots by behaviour, and its three behaviours line up with the groups here: Training with the training crawlers, Search with the AI search crawlers, Agent with the user-triggered fetchers. It applies to verified bots and to unverified bots that match them.

Block or allow by IP range

User agents can be faked; the address of a completed request can't. Where an operator publishes its crawler addresses, the tables link the official list. Use them in two ways: allow a crawler only from its own ranges, so a fake GPTBot from elsewhere gets nothing, or block those ranges outright.

Don't paste the ranges into your configuration once and forget them; operators change them. Fetch the lists on a schedule, as we do: our copies are refreshed every 6 hours, and the data sources page shows when each was last updated. For automation, our API answers the same question per address.

Check whether a bot is real

Paste an address from your logs, and the user agent if you have it, into Verify a bot. It checks the address against the official lists and, for operators that support it, forward-confirmed reverse DNS, then says whether the request really came from the bot it claims to be. The same method by hand, for Google, is in our guide to verifying a Googlebot IP.

Questions and answers

Will blocking GPTBot remove my site from ChatGPT search?

No. OpenAI uses OAI-SearchBot for search and GPTBot for training, and says Each setting is independent of the others. (source) To leave ChatGPT search answers you have to disallow OAI-SearchBot as well.

Does blocking Google-Extended hurt my Google ranking?

Google says Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search. (source) Apple says the same of its token: Site rules for Applebot-Extended are not considered in ranking for Search. (source)

Do AI crawlers obey robots.txt?

The documented crawlers in the tables say they do. Fetchers that act for a user are different: OpenAI writes that for ChatGPT-User robots.txt rules may not apply (source), and Perplexity that its user fetcher generally ignores robots.txt rules (source). Bots with no public documentation, such as Bytespider, have made no promise either way.

How long until a robots.txt change takes effect?

OpenAI says it can take ~24 hours from a site's robots.txt update for our systems to adjust. (source) The other operators in the tables don't publish a figure. A server or firewall rule applies to the next request.

Sources

Official documents we used to check the facts in this guide.