AI & bot directory · AI training
Bytespider: what it is, how to verify it and how to block it
Bytespider is run by ByteDance. ByteDance, TikTok's parent company, has published no documentation for it; community lists name it as a crawler that collects data for AI training.
Key facts
| Operator | ByteDance |
|---|---|
| Category | AI training |
| robots.txt name | Bytespider |
| Follows robots.txt | Couldn't check: Not documented |
| Can it be verified? | Couldn't check: No, user agent only |
| Official documentation | ByteDance has published none we could read. The user-agent name comes from the community lists ai.robots.txt and crawler-user-agents. |
| Registry checked | Nov 11, 2026 |
ByteDance publishes no documentation for Bytespider that we could read. What we know comes from community lists, so treat it as unconfirmed.
User agent
Its name in robots.txt is Bytespider.
Community lists recognise it by Bytespider in the user agent.
Can you verify Bytespider?
No No addresses are published, and there is no operator documentation, so any visitor can claim to be Bytespider. Treat its user agent as a claim.
Does Bytespider follow robots.txt?
Not documented We don't know: ByteDance hasn't said so anywhere we could read. robots.txt may or may not keep it out.
How to block or allow Bytespider
To ask it to stay away from the whole site, add this to robots.txt:
User-agent: Bytespider
Disallow: /
To keep it out of one folder only:
User-agent: Bytespider
Disallow: /private/
To allow it while asking every other bot to stay away:
User-agent: Bytespider
Allow: /
User-agent: *
Disallow: /
robots.txt is a request, not a lock. To enforce a block, add a firewall or CDN rule that matches Bytespider in the user agent. That stops the real Bytespider; a bot that lies about its user agent can still get through.
To build one robots.txt for several bots at once, use the robots.txt generator.
How often does Bytespider visit?
ByteDance doesn't publish how often Bytespider visits a site, and iplover doesn't see your traffic. Your own access log is the only reliable count: paste it into the log checker, which counts each crawler's visits and flags fakes, in your browser.
Official documentation
ByteDance has published none we could read. The user-agent name comes from the community lists ai.robots.txt and crawler-user-agents.
Registry checked Nov 11, 2026. The address lists are downloaded every 6 hours.