Check my connection

AI & bot directory · Other

Diffbot: what it is, how to verify it and how to block it

Diffbot is run by Diffbot. Diffbot has published no crawler documentation we could read; community lists describe it as extracting web data for its customers.

Key facts

Key facts
Operator
Diffbot
Category
Other
robots.txt name
Diffbot
Follows robots.txt
Couldn't check: Not documented
Can it be verified?
Couldn't check: No, user agent only
Official documentation
Diffbot has published none we could read. The user-agent name comes from the community lists ai.robots.txt and crawler-user-agents.
Registry checked
Nov 11, 2026

Diffbot publishes no documentation for Diffbot that we could read. What we know comes from community lists, so treat it as unconfirmed.

User agent

Its name in robots.txt is Diffbot.

Community lists recognise it by Diffbot in the user agent.

Can you verify Diffbot?

No No addresses are published, and there is no operator documentation, so any visitor can claim to be Diffbot. Treat its user agent as a claim.

Check an address from your logs

Does Diffbot follow robots.txt?

Not documented We don't know: Diffbot hasn't said so anywhere we could read. robots.txt may or may not keep it out.

How to block or allow Diffbot

To ask it to stay away from the whole site, add this to robots.txt:

User-agent: Diffbot
Disallow: /

To keep it out of one folder only:

User-agent: Diffbot
Disallow: /private/

To allow it while asking every other bot to stay away:

User-agent: Diffbot
Allow: /

User-agent: *
Disallow: /

robots.txt is a request, not a lock. To enforce a block, add a firewall or CDN rule that matches Diffbot in the user agent. That stops the real Diffbot; a bot that lies about its user agent can still get through.

To build one robots.txt for several bots at once, use the robots.txt generator.

How often does Diffbot visit?

Diffbot doesn't publish how often Diffbot visits a site, and iplover doesn't see your traffic. Your own access log is the only reliable count: paste it into the log checker, which counts each crawler's visits and flags fakes, in your browser.

Official documentation

Diffbot has published none we could read. The user-agent name comes from the community lists ai.robots.txt and crawler-user-agents.

Registry checked Nov 11, 2026. The address lists are downloaded every 6 hours.