AI crawlers, search bots, and training agents.

User-triggered fetches when ChatGPT opens a page to answer a person.

User-triggered fetches when Claude opens a page to answer a person.

User-triggered fetches for Perplexity answers and source citations.

Google user-triggered fetcher used by AI or product experiences.

Google fetcher associated with NotebookLM-style answer workflows.

Google fetcher associated with read-aloud and assistant experiences.

Google user-triggered fetcher used by AI or product experiences.

MistralAI-User is tracked as a user-triggered fetcher from Mistral.

User-triggered Microsoft/Copilot fetches for AI answers.

User-triggered Amazon fetches for fresh answers in products such as Alexa.

DuckDuckGo real-time crawler for AI-assisted answers with citations.

xAI-SearchBot is tracked as a user-triggered fetcher from xAI.

Grok-DeepSearch is tracked as a user-triggered fetcher from xAI.

Meta fetcher used when a person requests or shares a specific URL.

Kimi-User is tracked as a user-triggered fetcher from Moonshot AI.

Qwen-User is tracked as a user-triggered fetcher from Alibaba.

OpenAI crawler used to surface sites in ChatGPT search results.

Anthropic crawler used to discover and refresh content for Claude search.

Perplexity crawler used to keep its answer and search index fresh.

Google-InspectionTool is tracked as a search or answer-index crawler from Google.

Google crawler used for Search indexing and discovery. Cloudflare notes that Googlebot traffic can also support AI-related search features such as AI Overviews.

MistralAI-Index is tracked as a search or answer-index crawler from Mistral.

Microsoft crawler used for Bing Search indexing.

Legacy Microsoft crawler used for search indexing.

Amazon crawler used to make content eligible for Amazon search experiences.

Meta crawler used to improve Meta AI search result quality.

Kimi-SearchBot is tracked as a search or answer-index crawler from Moonshot AI.

TikTokSpider is tracked as a search or answer-index crawler from ByteDance.

Baiduspider is tracked as a search or answer-index crawler from Baidu.

YouBot is tracked as a search or answer-index crawler from You.com.

OpenAI crawler for collecting public content that may improve future models.

Anthropic crawler for public content that may be used to improve Claude.

Generic Google crawler used by product teams for public content fetches, including internal research and development.

Google crawler used for targeted crawls requested by site owners building Vertex AI agents.

Robots token controlling whether Apple may use crawled content for AI training.

Apple crawler Cloudflare classifies across search and training use cases.

Amazon crawler used to improve Amazon products and services.

Meta crawler for indexing or improving Meta products and AI systems.

KimiBot is tracked as a public-content crawler from Moonshot AI.

Bytespider is tracked as a public-content crawler from ByteDance.

ERNIEBot is tracked as a public-content crawler from Baidu.

QwenBot is tracked as a public-content crawler from Alibaba.

ChatGLM-Spider is tracked as a public-content crawler from Zhipu AI.

DeepSeekBot is tracked as a public-content crawler from DeepSeek.

cohere-ai is tracked as a public-content crawler from Cohere.

cohere-training-data-crawler is tracked as a public-content crawler from Cohere.

Allen Institute crawler used to find documents for AI research systems.

Common Crawl crawler that builds open web crawl datasets.

OpenAI crawler associated with ad or landing-page fetch workflows.

GrokBot is tracked as other ai bots from xAI.

xAI-Bot is tracked as other ai bots from xAI.

xAI-Grok is tracked as other ai bots from xAI.

xAI-Web-Crawler is tracked as other ai bots from xAI.

Grok is tracked as other ai bots from xAI.

Meta crawler used for advertising and business product use cases.

Meta crawler used for shared link previews on Facebook, Instagram, and Messenger.

Meta crawler traffic associated with Facebook bot fetches.

Doubaobot is tracked as other ai bots from ByteDance.

YiyanBot is tracked as other ai bots from Baidu.

TongyiBot is tracked as other ai bots from Alibaba.

AliyunBot is tracked as other ai bots from Alibaba.