AI Crawlers & Bots

HumanKey detects 19+ AI crawlers in real time. Updated as new bots emerge. See exactly who is crawling your site and why.

19+

Bots Detected

AI Companies

Real-time

Detection

GDPR

Compliant

Purpose:AI TrainingUser QueryAI SearchSearch IndexVoice AssistantApple Intelligence

Bot Name	Company	Purpose	Description	Status
GPTBotOpenAI	OpenAI	AI Training	OpenAI's primary web crawler for training data collection	Active
ChatGPT-UserOpenAI	OpenAI	User Query	Browses the web in response to real-time ChatGPT user queries	Active
OAI-SearchBotOpenAI	OpenAI	AI Search	Powers the search feature inside ChatGPT	Active
ClaudeBotAnthropic	Anthropic	AI Training	Anthropic's crawler for training Claude AI models	Active
Claude-WebAnthropic	Anthropic	User Query	Fetches URLs in real-time during Claude conversations	Active
Google-ExtendedGoogle	Google	AI Training	Google's opt-out crawler for Bard/Gemini training data	Active
GooglebotGoogle	Google	Search Index	Standard Google search indexing crawler	Active
PerplexityBotPerplexity AI	Perplexity AI	AI Search	Powers Perplexity's AI-powered search engine	Active
Meta-ExternalAgentMeta	Meta	AI Training	Meta's AI training crawler (+843% YoY growth in 2024)	Active
FacebookBotMeta	Meta	AI Training	Meta's legacy crawler for content indexing and AI training	Active
ApplebotApple	Apple	Voice Assistant	Powers Apple Siri web search and Spotlight suggestions	Active
Applebot-ExtendedApple	Apple	Apple Intelligence	Apple's opt-out crawler for Apple Intelligence AI training	Active
BingbotMicrosoft	Microsoft	Search Index	Microsoft Bing search index crawler, also powers Copilot	Active
BytespiderByteDance	ByteDance	AI Training	ByteDance (TikTok parent) crawler for AI model training	Active
AmazonbotAmazon	Amazon		Amazon's crawler for Alexa and Amazon AI services	Active
CCBotCommon Crawl	Common Crawl	AI Training	Open dataset used by many AI labs for model training	Active
cohere-aiCohere	Cohere	AI Training	Cohere's enterprise AI model training crawler	Active
AI2BotAI21 Labs / Allen AI	AI21 Labs / Allen AI	AI Training	Allen Institute for AI research crawler	Active
YouBotYou.com	You.com	AI Search	Powers You.com's AI-enhanced search engine	Active

⚠️ robots.txt bypass is growing

In Q1 2025, 12.9% of AI crawlers bypass robots.txt instructions (up from 3.3% in Q4 2024). Simply blocking bots in robots.txt is no longer sufficient — you need real-time detection to see what's actually happening.

Detect a bot we don't cover yet?

We add new AI crawlers continuously. Let us know which bot you're seeing and we'll add it to the detection database.

Report a Bot

Start detecting all these bots on your site — free, in under 5 minutes.

Start Free Trial →

No credit card required · GDPR-native · WordPress plugin available