Check my connection

AI & bot directory · Other

Google special-case crawlers: what it is, how to verify it and how to block it

Google special-case crawlers is run by Google. Google crawlers for specific products, such as checking the landing pages of ads, that work under an agreement with the site.

Key facts

Key facts
Operator
Google
Category
Other
robots.txt name
It has no single robots.txt name; Google lists several user agents in its documentation.
Follows robots.txt
Warning: In part
Can it be verified?
OK: Yes, by IP ranges and reverse DNS
Official documentation
Registry checked
Nov 11, 2026

User agent

It has no single robots.txt name; Google lists several user agents in its documentation.

Google doesn't publish a full user-agent string; look for Google special-case crawlers in the user agent.

Can you verify Google special-case crawlers?

Yes Google publishes the IP addresses Google special-case crawlers uses. We download that list every 6 hours: our copy is from Nov 20, 2026 and has 272 entries; Google dated it Nov 20, 2026.

Its addresses also have reverse-DNS names under google.com; a name only counts if it points back to the same address.

Check an address from your logs

Does Google special-case crawlers follow robots.txt?

In part Only in part: Google says Google special-case crawlers may not follow robots.txt in some cases, so robots.txt alone won't always keep it out.

“may ignore robots.txt rules”

How to block or allow Google special-case crawlers

It has no robots.txt name of its own; Google's documentation explains which user agents it uses.

robots.txt is a request, not a lock. To enforce a block, add a firewall or CDN rule. Because Google publishes Google special-case crawlers's addresses, a rule can match them exactly, so a fake Google special-case crawlers can't slip through as the real one.

How often does Google special-case crawlers visit?

Google doesn't publish how often Google special-case crawlers visits a site, and iplover doesn't see your traffic. Your own access log is the only reliable count: paste it into the log checker, which counts each crawler's visits and flags fakes, in your browser.

Official documentation

developers.google.com/crawling/docs/crawlers-fetchers/google-special-case-crawlers

Registry checked Nov 11, 2026. The address lists are downloaded every 6 hours.