AI & bot directory · Search engine
Claude-SearchBot: what it is, how to verify it and how to block it
Claude-SearchBot is run by Anthropic. Navigates the web to improve the quality of search results for Claude's users.
Key facts
| Operator | Anthropic |
|---|---|
| Category | Search engine |
| robots.txt name | Claude-SearchBot |
| Follows robots.txt | OK: Yes |
| Can it be verified? | OK: Yes, by IP ranges |
| Official documentation | |
| Registry checked | Nov 11, 2026 |
User agent
Its name in robots.txt is Claude-SearchBot.
Anthropic doesn't publish a full user-agent string; look for Claude-SearchBot in the user agent.
Can you verify Claude-SearchBot?
Yes Anthropic publishes the IP addresses Claude-SearchBot uses. We download that list every 6 hours: our copy is from Nov 15, 2026 and has 14 entries; Anthropic dated it Oct 20, 2026.
Anthropic publishes one list for all its bots, so the address proves the request came from Anthropic, but not which of its bots made it.
Does Claude-SearchBot follow robots.txt?
Yes Anthropic says Claude-SearchBot follows robots.txt.
“Anthropic's Bots respect "do not crawl" signals by honoring industry standard directives in robots.txt.”
How to block or allow Claude-SearchBot
To ask it to stay away from the whole site, add this to robots.txt:
User-agent: Claude-SearchBot
Disallow: /
To keep it out of one folder only:
User-agent: Claude-SearchBot
Disallow: /private/
To allow it while asking every other bot to stay away:
User-agent: Claude-SearchBot
Allow: /
User-agent: *
Disallow: /
robots.txt is a request, not a lock. To enforce a block, add a firewall or CDN rule. Because Anthropic publishes Claude-SearchBot's addresses, a rule can match them exactly, so a fake Claude-SearchBot can't slip through as the real one.
To build one robots.txt for several bots at once, use the robots.txt generator.
How often does Claude-SearchBot visit?
Anthropic doesn't publish how often Claude-SearchBot visits a site, and iplover doesn't see your traffic. Your own access log is the only reliable count: paste it into the log checker, which counts each crawler's visits and flags fakes, in your browser.
Official documentation
Registry checked Nov 11, 2026. The address lists are downloaded every 6 hours.