Google special-case crawlers: what it is, how to verify it and how to block it
Google special-case crawlers is run by Google. Google crawlers for specific products, such as checking the landing pages of ads, that work under an agreement with the site.
Key facts
| Operator | Google |
|---|---|
| Category | Other |
| robots.txt name | It has no single robots.txt name; Google lists several user agents in its documentation. |
| Follows robots.txt | Warning: In part |
| Can it be verified? | OK: Yes, by IP ranges and reverse DNS |
| Official documentation | |
| Registry checked | Nov 11, 2026 |
User agent
It has no single robots.txt name; Google lists several user agents in its documentation.
Google doesn't publish a full user-agent string; look for Google special-case crawlers in the user agent.
Can you verify Google special-case crawlers?
Yes Google publishes the IP addresses Google special-case crawlers uses. We download that list every 6 hours: our copy is from Nov 20, 2026 and has 272 entries; Google dated it Nov 20, 2026.
Its addresses also have reverse-DNS names under google.com; a name only counts if it points back to the same address.
Does Google special-case crawlers follow robots.txt?
In part Only in part: Google says Google special-case crawlers may not follow robots.txt in some cases, so robots.txt alone won't always keep it out.
“may ignore robots.txt rules”
How to block or allow Google special-case crawlers
It has no robots.txt name of its own; Google's documentation explains which user agents it uses.
robots.txt is a request, not a lock. To enforce a block, add a firewall or CDN rule. Because Google publishes Google special-case crawlers's addresses, a rule can match them exactly, so a fake Google special-case crawlers can't slip through as the real one.
How often does Google special-case crawlers visit?
Google doesn't publish how often Google special-case crawlers visits a site, and iplover doesn't see your traffic. Your own access log is the only reliable count: paste it into the log checker, which counts each crawler's visits and flags fakes, in your browser.
Official documentation
developers.google.com/crawling/docs/crawlers-fetchers/google-special-case-crawlers
Registry checked Nov 11, 2026. The address lists are downloaded every 6 hours.