Crawler & bot registry · reviewed September 2026
AI crawlers and search bots, from the operators' own documentation
109 documented crawlers and fetchers from 15 operators. For each one: the exact user-agent token, the robots.txt token, whether the operator says it honors robots.txt, how to verify a request is genuine, and the official IP range file where one exists. Nothing here is guessed; bots without official documentation are not listed.
- 109
- documented bots
- 15
- operators
- 46
- with official IP range files
- 12,153
- IP prefixes in those files
All crawlers
109 crawlers
| Name | User-agent token |
|---|---|
| AmazonbotAmazon | Amazonbot |
| ClaudeBotAnthropic | ClaudeBot |
| Applebot-ExtendedApple | Applebot-Extended |
| Google-ExtendedGoogle | Google-Extended |
| FacebookBotMeta | FacebookBot |
| Meta-ExternalAgentMeta | meta-externalagent |
| MistralAI-TrainingMistral AI | MistralAI-Training |
| GPTBotOpenAI | GPTBot |
| Amzn-SearchBotAmazon | Amzn-SearchBot |
| Claude-SearchBotAnthropic | Claude-SearchBot |
| DuckAssistBotDuckDuckGo | DuckAssistBot |
| Google-CloudVertexBotGoogle | Google-CloudVertexBot |
| Meta-WebIndexerMeta | meta-webindexer |
| MistralAI-IndexMistral AI | MistralAI-Index |
| OAI-SearchBotOpenAI | OAI-SearchBot |
| PerplexityBotPerplexity | PerplexityBot |
| YandexAdditionalBotYandex | YandexAdditionalBot |
| Amzn-UserAmazon | Amzn-User |
| Claude-UserAnthropic | Claude-User |
| Chrome Web StoreGoogle | Google-CWS |
| FeedfetcherGoogle | FeedFetcher-Google |
| Gemini NotebookGoogle | Google-GeminiNotebook |
| Google PinpointGoogle | Google-Pinpoint |
| Google Publisher CenterGoogle | GoogleProducer |
| Google Read AloudGoogle | Google-Read-Aloud |
| Google Site VerifierGoogle | Google-Site-Verification |
| Google-AgentGoogle | Google-Agent |
| Meta-ExternalFetcherMeta | meta-externalfetcher |
| MistralAI-UserMistral AI | MistralAI-User |
| ChatGPT-UserOpenAI | ChatGPT-User |
| Perplexity-UserPerplexity | Perplexity-User |
| YandexCalendarYandex | YandexCalendar |
| YandexDialogsYandex | YandexDialogs |
| YandexFeedYandex | YandexFeed |
| YandexSearchShopYandex | YandexSearchShop |
| YandexUserproxyYandex | YandexUserproxy |
| ApplebotApple | Applebot |
| BytespiderByteDance | Bytespider |
| DuckDuckBotDuckDuckGo | DuckDuckBot |
| Google StoreBotGoogle | Storebot-Google |
| GooglebotGoogle | Googlebot |
| Googlebot ImageGoogle | Googlebot-Image |
| Googlebot NewsGoogle | Googlebot-News |
| Googlebot VideoGoogle | Googlebot-Video |
| BingVideoPreviewMicrosoft Bing | BingVideoPreview/1.0 |
| BingbotMicrosoft Bing | bingbot/2.0 |
| YaComBotYandex | YaComBot |
| YandexBlogsYandex | YandexBlogs |
| YandexBotYandex | YandexBot |
| YandexComBotYandex | YandexComBot |
| YandexFaviconsYandex | YandexFavicons |
| YandexImagesYandex | YandexImages |
| YandexMediaYandex | YandexMedia |
| YandexMobileBotYandex | YandexMobileBot |
| YandexOntoDBYandex | YandexOntoDB |
| YandexRenderResourcesBotYandex | YandexRenderResourcesBot |
| YandexSitelinksYandex | YandexSitelinks |
| YandexVerticalsYandex | YandexVerticals |
| YandexVertisYandex | YandexVertis |
| YandexVideoYandex | YandexVideo |
| YandexVideoParserYandex | YandexVideoParser |
| Google MessagesGoogle | GoogleMessages |
| FacebookExternalHitMeta | facebookexternalhit |
| MicrosoftPreviewMicrosoft Bing | MicrosoftPreview/2.0 |
| AhrefsBotAhrefs Pte. Ltd. | AhrefsBot |
| AhrefsSiteAuditAhrefs Pte. Ltd. | AhrefsSiteAudit |
| Google-InspectionToolGoogle | Google-InspectionTool |
| RyteBotSemrush | RyteBot |
| SemrushBotSemrush | SemrushBot |
| SemrushBot-BASemrush | SemrushBot-BA |
| SemrushBot-ESISemrush | SemrushBot-ESI |
| SemrushBot-FTSemrush | SemrushBot-FT |
| SemrushBot-OCOBSemrush | SemrushBot-OCOB |
| SemrushBot-SISemrush | SemrushBot-SI |
| SemrushBot-SWASemrush | SemrushBot-SWA |
| SiteAuditBotSemrush | SiteAuditBot |
| SplitSignalBotSemrush | SplitSignalBot |
| YandexPagecheckerYandex | YandexPagechecker |
| YandexWebmasterYandex | YandexWebmaster |
| CCBotCommon Crawl | CCBot |
| iTMSApple | iTMS |
| APIs-GoogleGoogle | APIs-Google |
| AdSenseGoogle | Mediapartners-Google |
| AdsBotGoogle | AdsBot-Google |
| AdsBot Mobile WebGoogle | AdsBot-Google-Mobile |
| Google-SafetyGoogle | Google-Safety |
| GoogleOtherGoogle | GoogleOther |
| GoogleOther-ImageGoogle | GoogleOther-Image |
| GoogleOther-VideoGoogle | GoogleOther-Video |
| Meta-ExternalAdsMeta | meta-externalads |
| AdIdxBotMicrosoft Bing | adidxbot/2.0 |
| OAI-AdsBotOpenAI | OAI-AdsBot |
| YaDirectFetcherYandex | YaDirectFetcher |
| YandexYandex | Yandex |
| YandexAccessibilityBotYandex | YandexAccessibilityBot |
| YandexAdNetYandex | YandexAdNet |
| YandexCheckBotYandex | YandexCheckBot |
| YandexDirectYandex | YandexDirect |
| YandexDirectDynYandex | YandexDirectDyn |
| YandexImageResizerYandex | YandexImageResizer |
| YandexMarketYandex | YandexMarket |
| YandexMetrikaYandex | YandexMetrika |
| YandexMobileScreenShotBotYandex | YandexMobileScreenShotBot |
| YandexOntoDBAPIYandex | YandexOntoDBAPI |
| YandexPartnerYandex | YandexPartner |
| YandexRCAYandex | YandexRCA |
| YandexScreenshotBotYandex | YandexScreenshotBot |
| YandexSpravBotYandex | YandexSpravBot |
| YandexTrackerYandex | YandexTracker |
Crawler types
- AI training crawlers8 bots
- Collects public web content that may be used to train or improve AI models. These are the tokens most site owners decide about first. Amazonbot, ClaudeBot, Applebot-Extended, Google-Extended, FacebookBot, Meta-ExternalAgent and more.
- AI search indexs9 bots
- Builds an index that an AI assistant searches at answer time, so it can cite and link pages. Blocking it removes a site from those answers. Amzn-SearchBot, Claude-SearchBot, DuckAssistBot, Google-CloudVertexBot, Meta-WebIndexer, MistralAI-Index and more.
- User-triggered fetchers19 bots
- Fetches a page because a person asked an assistant or app to open it. Operators generally treat these like a browser acting on the user's behalf. Amzn-User, Claude-User, Chrome Web Store, Feedfetcher, Gemini Notebook, Google Pinpoint and more.
- Search engine crawlers25 bots
- Classic web search indexing. Blocking it removes a site from that search engine. Applebot, Bytespider, DuckDuckBot, Google StoreBot, Googlebot, Googlebot Image and more.
- SEO tool crawlers15 bots
- Crawls to build backlink and site-audit indexes sold to marketers. AhrefsBot, AhrefsSiteAudit, Google-InspectionTool, RyteBot, SemrushBot, SemrushBot-BA and more.
- Archive crawlers1 bots
- Crawls to build a public archive or open corpus of the web. CCBot.
- Others29 bots
- Documented bots that do not fit the categories above. iTMS, APIs-Google, AdSense, AdsBot, AdsBot Mobile Web, Google-Safety and more.
By operator
- Yandex40 bots: YandexAdditionalBot, YandexCalendar, YandexDialogs, YandexFeed, YandexSearchShop, YandexUserproxy, YaComBot, YandexBlogs, YandexBot, YandexComBot, YandexFavicons, YandexImages, YandexMedia, YandexMobileBot, YandexOntoDB, YandexRenderResourcesBot, YandexSitelinks, YandexVerticals, YandexVertis, YandexVideo, YandexVideoParser, YandexPagechecker, YandexWebmaster, YaDirectFetcher, Yandex, YandexAccessibilityBot, YandexAdNet, YandexCheckBot, YandexDirect, YandexDirectDyn, YandexImageResizer, YandexMarket, YandexMetrika, YandexMobileScreenShotBot, YandexOntoDBAPI, YandexPartner, YandexRCA, YandexScreenshotBot, YandexSpravBot, YandexTracker
- Google25 bots: Google-Extended, Google-CloudVertexBot, Chrome Web Store, Feedfetcher, Gemini Notebook, Google Pinpoint, Google Publisher Center, Google Read Aloud, Google Site Verifier, Google-Agent, Google StoreBot, Googlebot, Googlebot Image, Googlebot News, Googlebot Video, Google Messages, Google-InspectionTool, APIs-Google, AdSense, AdsBot, AdsBot Mobile Web, Google-Safety, GoogleOther, GoogleOther-Image, GoogleOther-Video
- Semrush10 bots: RyteBot, SemrushBot, SemrushBot-BA, SemrushBot-ESI, SemrushBot-FT, SemrushBot-OCOB, SemrushBot-SI, SemrushBot-SWA, SiteAuditBot, SplitSignalBot
- Meta6 bots: FacebookBot, Meta-ExternalAgent, Meta-WebIndexer, Meta-ExternalFetcher, FacebookExternalHit, Meta-ExternalAds
- Microsoft Bing4 bots: BingVideoPreview, Bingbot, MicrosoftPreview, AdIdxBot
- OpenAI4 bots: GPTBot, OAI-SearchBot, ChatGPT-User, OAI-AdsBot
- Amazon3 bots: Amazonbot, Amzn-SearchBot, Amzn-User
- Anthropic3 bots: ClaudeBot, Claude-SearchBot, Claude-User
- Apple3 bots: Applebot-Extended, Applebot, iTMS
- Mistral AI3 bots: MistralAI-Training, MistralAI-Index, MistralAI-User
- Ahrefs Pte. Ltd.2 bots: AhrefsBot, AhrefsSiteAudit
- DuckDuckGo2 bots: DuckAssistBot, DuckDuckBot
- Perplexity2 bots: PerplexityBot, Perplexity-User
- ByteDance1 bot: Bytespider
- Common Crawl1 bot: CCBot
How this registry is built
Each entry was compiled from the operator's own documentation, then handed to a second reviewer whose only job was to re-open every source and strike any claim not found there. What survived is what you see. Fields the operator does not document are marked as such rather than filled in.
IP-range figures come from a script that downloads each operator's official range file, counts the prefixes, and records the file's stated creation time and a SHA-256 of the body. The last run was 2026-09-06. Every bot page links the live file so you can compare.
The registry is read-only reference. It documents what operators say about their own bots so site owners can identify and control them; it does not publish third-party IP lists or speculate about undocumented behaviour.
Common questions
- What is the crawler and bot registry?
- A read-only reference of 109 web crawlers and fetchers from 15 operators, including the AI crawlers site owners ask about most: GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot and Googlebot. Each entry gives the exact user-agent token, the robots.txt token, whether the operator says it honors robots.txt, the documented way to verify a request, and the official IP range file where one is published. Reviewed September 2026.
- Where does the data come from?
- Only from each operator's own documentation and official machine-readable range files. Every field was checked against the operator's page and re-checked by an independent reviewer. IP-range counts are produced by a script that downloads the operator's file and records its creation time and a hash, last run 2026-09-06. Bots without official documentation are not listed.
- How do I block AI crawlers from training on my site?
- Add a robots.txt rule per token. Training crawlers documented here include GPTBot, ClaudeBot and CCBot; Google uses the policy token Google-Extended, which has no crawler of its own. Each entry has copy-ready allow and block blocks. Note that user-triggered fetchers such as ChatGPT-User act on a person's request and some operators document that they do not consult robots.txt.
- How can I tell whether a request is really from the bot it claims to be?
- Anyone can send any User-Agent string. The two documented checks are a reverse DNS lookup (the IP must resolve to the operator's domain and back again) and a match against the operator's published IP ranges. Each entry states which of those the operator supports.
- Is the dataset free to use?
- Yes. The full registry is published as an open dataset under CC BY 4.0, downloadable as CSV, NDJSON or JSON, with citation blocks on the dataset page.