Training crawlersUser agent + IP range

Google

GoogleOther

About this crawler

Generic Google crawler used by product teams for public content fetches, including internal research and development.

Usually means: The crawler is collecting public web content that may be used for model training, datasets, or product improvement.

User agent

GoogleOther

DataFast matches these as user-agent tokens, not full browser strings. Providers may add surrounding version or product text.

IP verification

Verified when user agent and IP range both match

DataFast first matches the crawler user-agent token, then compares the request IP with published ranges from the crawler operator when those ranges exist.

User agent

Request user agent contains GoogleOther.

Request IP

Server-side tracking forwards the crawler IP seen by your backend.

Range check

The IP is compared with published CIDR ranges or JSON range endpoints.

Sample IP ranges

2001:4860:4801:10::/642001:4860:4801:11::/642001:4860:4801:12::/642001:4860:4801:13::/642001:4860:4801:14::/642001:4860:4801:15::/642001:4860:4801:16::/642001:4860:4801:17::/642001:4860:4801:18::/642001:4860:4801:19::/64

Showing up to 10 ranges from the configured official sources.

Range endpoints

https://developers.google.com/static/crawling/ipranges/common-crawlers.json https://developers.google.com/static/crawling/ipranges/special-crawlers.json https://developers.google.com/static/crawling/ipranges/user-triggered-fetchers.json https://developers.google.com/static/crawling/ipranges/user-triggered-fetchers-google.json https://developers.google.com/static/crawling/ipranges/user-triggered-agents.json

Discover when bots and AI assistants crawl your website.

Install the server-side tracker to see which AI assistants, search engines, and training crawlers request your pages. It helps you understand what bots are trying to find, including pages that return 404.

Set up bot tracking npm package