How to verify a web crawler is real: Googlebot, Bingbot and spotting fake bots
A request in your log says Googlebot. Maybe it is Google; maybe it's a scraper that copied Google's name because few site owners dare to block it. This guide shows the two checks that actually prove who sent a request, for the major search engine crawlers and the AI search crawlers, with each operator's official IP list, its reverse DNS names and rules to refuse the fakes.
"Last updated" at the top is when this article was last revised. "Registry checked", under each table, is when we last checked every crawler in our registry against its operator's documentation; the official IP lists are downloaded again every 6 hours.
Why the user agent proves nothing
Think of the user agent as a name badge the visitor writes for themselves. Real crawlers write an honest one, but nothing stops anyone else from writing Googlebot on theirs, and copying Googlebot's exact user agent string takes one line of code. Scrapers and attack tools do it on purpose: some firewalls and rate limits wave known crawlers through.
The address is different. It is where the connection really came from, and only the operator controls the addresses it publishes and the DNS names under its own domain. Both checks below use the address, never the badge.
The two checks that work
You need one of these to pass, whichever the operator supports. Some operators support both.
1. The operator's published IP list
Many operators publish the address ranges their crawler uses, usually as a JSON file of IPv4 and IPv6 prefixes. If the address falls inside one of them, the request came from that operator's network. This is the quickest check, and the one to use in a firewall, because it needs no lookup per request. Lists change, so download them again regularly instead of copying them once.
2. Forward-confirmed reverse DNS (FCrDNS)
This is the method Google documents for its own crawlers. It takes two DNS lookups:
- Reverse lookup: ask DNS for the name of the address (its
PTRrecord). - Check the name ends in the operator's domain. For Google:
Verify that the domain name is either googlebot.com, google.com, or googleusercontent.com.
(source) - Forward lookup: ask DNS for the address of that name.
- Confirm it matches. Google:
Verify that it's the same as the original accessing IP address from your logs.
(source)
Step 3 is what makes this proof. Whoever controls an address can set its reverse name to anything, including crawl-1-2-3-4.googlebot.com. Only Google can make a name under googlebot.com point back to an address. With dig on Linux or macOS, Google's own example looks like this:
$ dig -x 66.249.66.1 +short
crawl-66-249-66-1.googlebot.com.
$ dig +short crawl-66-249-66-1.googlebot.com
66.249.66.1
On Windows, nslookup does the same two lookups:
> nslookup 66.249.66.1
> nslookup crawl-66-249-66-1.googlebot.com
Bingbot addresses have reverse names under search.msn.com that resolve back the same way; this is a real lookup of a Bingbot address. We verify Bingbot against Bing's published list, linked in the table below.
$ dig -x 40.77.167.10 +short
msnbot-40-77-167-10.search.msn.com.
$ dig +short msnbot-40-77-167-10.search.msn.com
40.77.167.10
A tool that shows only the reverse name (the PTR record) is showing step 1, which anyone can fake. Our Verify a bot tool does both lookups and checks the official lists; our guide to reverse DNS explains PTR records in general.
Is this crawler real? A quick checklist
| Check | Real crawler | Fake |
|---|---|---|
| The user agent names a known crawler | Yes | Usually yes too: this proves nothing |
| The address is in the operator's published list | Yes | No |
| The reverse DNS name is under the operator's domain | Yes, for operators that use it | Missing, generic, or another domain |
| That name resolves back to the same address | Yes | No, or not at all |
| What it requests | Pages and robots.txt, at a steady pace | Often login pages, admin paths or search forms: a hint, not proof |
The major crawlers and how to verify each
For each crawler: the user-agent token, how a request can be proven to come from it, the reverse DNS names where the operator uses them, and the official sources. Each name links to the crawler's page in our AI and bot directory. The tables cover the crawlers in our registry, not every bot on the web.
Search engine crawlers
| Bot | Operator | robots.txt name | Can it be verified? | Reverse DNS name ends in | Sources |
|---|---|---|---|---|---|
| Googlebot | Googlebot |
Yes, by IP ranges and reverse DNS | .googlebot.com .geo.googlebot.com |
Official IP list: developers.google.com/static/crawling/ipranges/common-crawlers.jsonOperator's documentation: developers.google.com |
|
| Bingbot | Microsoft | bingbot |
Yes, by IP ranges | – |
Official IP list: bing.com/toolbox/bingbot.jsonOperator's documentation: bing.com |
| Applebot | Apple | Applebot |
Yes, by IP ranges and reverse DNS | .applebot.apple.com |
Official IP list: search.developer.apple.com/applebot.jsonOperator's documentation: support.apple.com |
| DuckDuckBot | DuckDuckGo | DuckDuckBot |
Yes, by IP ranges | – |
Official IP list: duckduckgo.com/duckduckbot.jsonOperator's documentation: duckduckgo.com |
Google runs more than Googlebot: special-case crawlers and user-triggered fetchers come from separate published lists and have reverse names under google.com or gae.googleusercontent.com rather than googlebot.com. Check an address against the list for the crawler type it claims to be.
AI search crawlers
| Bot | Operator | robots.txt name | Can it be verified? | Reverse DNS name ends in | Sources |
|---|---|---|---|---|---|
| OAI-SearchBot | OpenAI | OAI-SearchBot |
Yes, by IP ranges | – |
Official IP list: openai.com/searchbot.jsonOperator's documentation: developers.openai.com |
| PerplexityBot | Perplexity | PerplexityBot |
Yes, by IP ranges | – |
Official IP list: perplexity.ai/perplexitybot.jsonOperator's documentation: docs.perplexity.ai |
| Claude-SearchBot | Anthropic | Claude-SearchBot |
Yes, by IP ranges | – |
Official IP list: claude.com/crawling/bots.jsonOperator's documentation: support.claude.com |
| Amzn-SearchBot | Amazon | Amzn-SearchBot |
Yes, by IP ranges | – |
Official IP list: developer.amazon.com/amazonbot/searchbot-ip-addresses/Operator's documentation: developer.amazon.com |
| meta-webindexer | Meta | meta-webindexer |
No, user agent only | – |
Operator's documentation: developers.facebook.com |
These index pages for AI answer engines. To decide which AI bots to let in at all, see our guide on how to block AI crawlers. A crawler marked "user agent only" publishes nothing to check against, so a request that names it can't be proven either way.
How to spot a fake crawler in your logs
Start from the requests that name a crawler, then test their addresses. A fake usually gives itself away in one of these ways:
- The address fails the checks above: it's in no published list, has no reverse name under the operator's domain, or its name doesn't resolve back.
- One name, many unrelated networks: real Googlebot traffic comes from Google's ranges, not from a dozen hosting companies at once.
- Its behaviour doesn't fit: hammering login or admin URLs, trying every search parameter, or ignoring your robots.txt. Treat this as a reason to check, not as proof on its own.
One trap first: behind a CDN or reverse proxy, your server may log the proxy's address instead of the visitor's. Check that you are testing the real client address (for example CF-Connecting-IP behind Cloudflare, or the address your proxy passes on), or every crawler will look fake.
Our log checker does this for a whole log in your browser: it lists each address, what its user agent claims, and whether a crawler claim can be confirmed.
What to do about fake crawlers
Refuse requests that name a crawler but don't come from its addresses. The rules below do it for Googlebot with Google's published list; the same pattern works for any crawler that publishes one, such as Bingbot with Bing's list. Real Googlebot and ordinary visitors are unaffected.
nginx
The geo block marks addresses from Google's list, the first map marks user agents that say Googlebot, and the second map combines them: a request that claims Googlebot from outside the list gets 403.
# In the http { } block.
# 1. Is the address in Google's published crawler list? (file made by the script below)
geo $googlebot_ip {
default 0;
include /etc/nginx/googlebot-ips.conf;
}
# 2. Does the user agent say Googlebot?
map $http_user_agent $claims_googlebot {
default 0;
"~*Googlebot" 1;
}
# 3. Says Googlebot, but isn't from Google's list: a fake
map "$claims_googlebot:$googlebot_ip" $fake_googlebot {
default 0;
"1:0" 1;
}
# In each server { } block:
if ($fake_googlebot) { return 403; }
The list file is made, and kept fresh, by a small script. Run it daily; it refuses to install an empty list, which would turn every real Googlebot away, and reloads nginx only if the configuration tests clean (nginx -t).
#!/bin/sh
# Refresh Google's crawler addresses for nginx. Run it daily (cron, systemd timer).
# Needs curl and jq. Writes one "CIDR 1;" line per range, e.g. "192.0.2.0/24 1;".
set -e
curl -fsS https://developers.google.com/static/crawling/ipranges/common-crawlers.json \
| jq -r '.prefixes[] | (.ipv4Prefix // .ipv6Prefix) + " 1;"' > /etc/nginx/googlebot-ips.conf.new
# Never install an empty list: it would turn every real Googlebot away.
[ -s /etc/nginx/googlebot-ips.conf.new ]
mv /etc/nginx/googlebot-ips.conf.new /etc/nginx/googlebot-ips.conf
nginx -t && nginx -s reload
Apache
In Apache, an block applies only to requests whose user agent says Googlebot, and its Require ip line allows only Google's ranges. The script below writes that block from Google's list.
# In <VirtualHost> or <Directory> (Apache 2.4, mod_authz_core):
# a request that says Googlebot must come from Google's published addresses.
Include /etc/apache2/googlebot-require.conf
#!/bin/sh
# Refresh the Googlebot rule for Apache. Run it daily. Needs curl and jq.
set -e
F=/etc/apache2/googlebot-require.conf
{
echo '<If "%{HTTP_USER_AGENT} =~ /googlebot/i">'
curl -fsS https://developers.google.com/static/crawling/ipranges/common-crawlers.json \
| jq -r '" Require ip " + ([.prefixes[] | (.ipv4Prefix // .ipv6Prefix)] | join(" "))'
echo '</If>'
} > "$F.new"
grep -q 'Require ip [0-9a-f]' "$F.new" # never install an empty rule
mv "$F.new" "$F"
apachectl configtest && apachectl graceful
Cloudflare
On Cloudflare, a WAF custom rule with the action Block can use Cloudflare's own list of good bots: the field cf.client.bot is true for requests from known good bots and crawlers, so a request that names a search engine crawler without it is refused.
(http.user_agent contains "Googlebot" or http.user_agent contains "bingbot" or http.user_agent contains "Applebot" or http.user_agent contains "DuckDuckBot")
and not cf.client.bot
Blocking fakes doesn't stop real crawlers, and it shouldn't: refusing the real Googlebot drops your pages from Google Search. If you want to keep AI crawlers out as well, that's a separate decision, covered in how to block AI crawlers.
Automate the check
Google's advice for volume is to automate: For large scale lookups, use an automatic solution
(source) against the published lists. Our API answers the same question as Verify a bot for one address and user agent at a time, and the data sources page shows when each official list was last downloaded.
Looking for a specific crawler? The AI & bot directory lists each one's user agent, IP ranges and robots.txt name. To block several AI bots at once, use the robots.txt generator.
Questions and answers
Is a reverse DNS (PTR) record enough?
No. Whoever controls an address block sets its reverse DNS names, so a PTR record can claim any hostname. Only the forward lookup, done under the operator's own domain, ties the name to the address.
Can a real crawler fail the check?
Yes, in a few honest ways: the operator added addresses after your copy of its list was downloaded, your server logs a proxy's or CDN's address instead of the visitor's, or a DNS lookup timed out. Refresh lists on a schedule, log the real client address, and treat a single DNS failure as unknown, not fake.
Should I run DNS lookups on every request?
No. Lookups add delay and can fail under load. Match addresses against the published lists in your server or firewall, and keep reverse DNS for spot checks and for crawlers without a list, caching each answer.
Can every crawler be verified?
No. Some operators publish addresses, some only reverse DNS names, some both, and a few neither. The tables above say which is which; a crawler you can't verify can only be judged by what it does.
Sources
Official documents we used to check the facts in this guide.
- Google Search Central: Verify requests from Google crawlers and fetchers
- Google Search Central: Google's common crawlers
- Bing Webmaster Tools: Which crawlers does Bing use?
- Bing: bingbot.json (published Bingbot addresses)
- Apple: About Applebot
- DuckDuckGo: DuckDuckBot
- OpenAI: Overview of OpenAI crawlers
- Perplexity: Perplexity crawlers
- Anthropic: Does Anthropic crawl data from the web?
- nginx: ngx_http_geo_module
- nginx: ngx_http_map_module
- Apache HTTP Server 2.4: mod_authz_core (Require ip)
- Apache HTTP Server 2.4: Expressions (<If>)
- Cloudflare: cf.client.bot