About
Bytespider is ByteDance's web crawler, collecting training data for its AI models including Doubao. It is known for aggressive crawl volumes and has been reported to ignore robots.txt at times.
ByteDance / Doubao training
Details
- Type
- AI crawler
- Operator
- ByteDance
- Access
- Direct — crawls on the operator’s own schedule
- Traffic rank