About
Generic crawler that may be used by various product teams for fetching publicly accessible content from sites. For example, it may be used for one-off crawls for internal research and development. https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers#googleother
Internal Google crawls, plus -image and -video variants
Details
- Type
- AI crawler
- Operator
- Access
- Direct — crawls on the operator’s own schedule
- Traffic rank
- #18