AI & bot directory · Search engine
meta-webindexer: what it is, how to verify it and how to block it
meta-webindexer is run by Meta. Indexes web pages for search in Meta AI.
Key facts
| Operator | Meta |
|---|---|
| Category | Search engine |
| robots.txt name | meta-webindexer |
| Follows robots.txt | OK: Yes |
| Can it be verified? | Couldn't check: No, user agent only |
| Official documentation | |
| Registry checked | Nov 11, 2026 |
User agent
Its name in robots.txt is meta-webindexer.
Meta documents this user-agent string:
meta-webindexer/1.1
Can you verify meta-webindexer?
No Meta publishes no IP addresses or reverse-DNS names for meta-webindexer, so any visitor can claim to be it. Treat its user agent as a claim.
Does meta-webindexer follow robots.txt?
Yes Meta says meta-webindexer follows robots.txt.
“In order to block these crawlers, add a disallow for the relevant crawler to robots.txt.”
How to block or allow meta-webindexer
To ask it to stay away from the whole site, add this to robots.txt:
User-agent: meta-webindexer
Disallow: /
To keep it out of one folder only:
User-agent: meta-webindexer
Disallow: /private/
To allow it while asking every other bot to stay away:
User-agent: meta-webindexer
Allow: /
User-agent: *
Disallow: /
robots.txt is a request, not a lock. To enforce a block, add a firewall or CDN rule that matches meta-webindexer in the user agent. That stops the real meta-webindexer; a bot that lies about its user agent can still get through.
To build one robots.txt for several bots at once, use the robots.txt generator.
How often does meta-webindexer visit?
Meta doesn't publish how often meta-webindexer visits a site, and iplover doesn't see your traffic. Your own access log is the only reliable count: paste it into the log checker, which counts each crawler's visits and flags fakes, in your browser.
Official documentation
developers.facebook.com/docs/sharing/webmasters/web-crawlers/
Registry checked Nov 11, 2026. The address lists are downloaded every 6 hours.