AI & bot directory · AI assistant fetch
meta-externalfetcher: what it is, how to verify it and how to block it
meta-externalfetcher is run by Meta. Fetches individual links at a user's request, for example for Meta's AI assistant features.
Key facts
| Operator | Meta |
|---|---|
| Category | AI assistant fetch |
| robots.txt name | meta-externalfetcher |
| Follows robots.txt | Warning: In part |
| Can it be verified? | Couldn't check: No, user agent only |
| Official documentation | |
| Registry checked | Nov 11, 2026 |
User agent
Its name in robots.txt is meta-externalfetcher.
Meta documents this user-agent string:
meta-externalfetcher/1.1
Can you verify meta-externalfetcher?
No Meta publishes no IP addresses or reverse-DNS names for meta-externalfetcher, so any visitor can claim to be it. Treat its user agent as a claim.
Does meta-externalfetcher follow robots.txt?
In part Only in part: Meta says meta-externalfetcher may not follow robots.txt in some cases, so robots.txt alone won't always keep it out.
“this crawler may bypass robots.txt rules”
How to block or allow meta-externalfetcher
To ask it to stay away from the whole site, add this to robots.txt:
User-agent: meta-externalfetcher
Disallow: /
To keep it out of one folder only:
User-agent: meta-externalfetcher
Disallow: /private/
To allow it while asking every other bot to stay away:
User-agent: meta-externalfetcher
Allow: /
User-agent: *
Disallow: /
robots.txt is a request, not a lock. To enforce a block, add a firewall or CDN rule that matches meta-externalfetcher in the user agent. That stops the real meta-externalfetcher; a bot that lies about its user agent can still get through.
To build one robots.txt for several bots at once, use the robots.txt generator.
How often does meta-externalfetcher visit?
Meta doesn't publish how often meta-externalfetcher visits a site, and iplover doesn't see your traffic. Your own access log is the only reliable count: paste it into the log checker, which counts each crawler's visits and flags fakes, in your browser.
Official documentation
developers.facebook.com/docs/sharing/webmasters/web-crawlers/
Registry checked Nov 11, 2026. The address lists are downloaded every 6 hours.