Check my connection

robots.txt generator

Tick the bots to block (or the only ones to allow) from our directory of 32 crawlers and AI bots. You get a robots.txt to copy, and the matching Cloudflare, nginx and Apache rules to enforce it.

Start from a preset

What should the ticked bots do?
Search engine
AI training
AI assistant fetch
SEO crawler
Other

Your rules

robots.txt

# Generated by iplover.com/robots-txt-generator on 2026-11-21
# robots.txt is a request: well-behaved bots follow it, others can ignore it.

User-agent: Claude-SearchBot
Disallow: /

User-agent: *
Allow: /

Cloudflare (WAF custom rule)

# Cloudflare dashboard > Security > WAF > Custom rules > Create rule > Edit expression:

(lower(http.user_agent) contains "claude-searchbot")

# Action: Block

nginx

# Inside your server { } block
if ($http_user_agent ~* "(Claude-SearchBot)") {
    return 403;
}

Apache (.htaccess)

# .htaccess (needs mod_rewrite)
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} (Claude-SearchBot) [NC]
RewriteRule .* - [F,L]

robots.txt is a request that well-behaved bots follow. It does not stop a bot that ignores it, and some AI fetchers ignore it when a person asked for the page. To enforce a block, use the server rules above or your firewall or CDN. They match the user agent, which a dishonest bot can fake.

Registry checked 2026-11-11. The address lists are downloaded every 6 hours.

What this tool does

robots.txt is a plain-text file at the root of your site (example.com/robots.txt) that tells crawlers which parts they may fetch. Each bot looks for its own name in a “User-agent” line and follows the rules under it.

AI companies now run several kinds of bots: some collect pages to train models, some fetch a page when a person asks an assistant about it, and some build search indexes. Blocking the training bots doesn't remove you from normal search results; blocking search crawlers does.

Every bot here, with its exact name, operator and what it does, is in the AI & bot directory.

How to read the results

  • Pick a preset or tick bots, then choose whether the ticked ones are blocked or are the only ones allowed.
  • Copy the robots.txt and put it at the root of your site, replacing (or merging with) the one you have. Rules for each bot apply to the whole site (Disallow: /).
  • To enforce the block, add the Cloudflare, nginx or Apache rule. It answers matching user agents with “403 Forbidden”.
  • Check that it works: paste your access log into the log checker after a few days, or verify a visitor that claims to be one of these bots.

Frequently asked questions

How do I block ChatGPT and other AI bots from my site?

Use the preset “Block all AI bots, keep search”. It adds GPTBot (training) and ChatGPT-User (fetches for users) and the equivalents from other AI companies to robots.txt. OpenAI says GPTBot follows robots.txt; ChatGPT-User may not, because a person asked for the page, so add the server rule too.

Will blocking AI bots hurt my Google ranking?

Not if you keep the search crawlers. Googlebot is what indexes your pages for Google Search. Google-Extended is a separate robots.txt control: blocking it stops your content being used for Gemini training without affecting Search.

Does robots.txt stop all bots?

No. It is a request. Reputable crawlers follow it, but scrapers and some AI fetchers ignore it. For bots that ignore robots.txt, use a firewall or CDN rule, and remember that a bot can fake its user agent.

What is the difference between blocking and allowing only some bots?

“Block” lists the bots to keep out and lets everyone else in. “Allow only” lets the listed bots in and asks every other bot to stay away, including ones that don't exist yet; that is stricter, and new search engines will be kept out too.