Open datasetCC BY 4.0 · reviewed September 2026
Crawler & Bot Registry dataset
109 documented crawlers and fetchers from 15 operators, one row each: user-agent token, robots.txt token, whether the operator says it honors robots.txt, the official IP range file with its prefix count and creation time, the documented verification method, and the source documentation URL. Free to use with attribution.
109
documented bots
46
with official IP range files
12,153
IP prefixes across those files
55/109
documented as honoring robots.txt
By purpose
Every row
| Bot | Operator | User-agent token | robots.txt token | Honors robots | IP prefixes | Verification |
|---|---|---|---|---|---|---|
| Amazonbot | Amazon | Amazonbot | Amazonbot | yes | 525 | published ranges |
| ClaudeBot | Anthropic | ClaudeBot | ClaudeBot | yes | 26 | published ranges |
| Applebot-Extended | Apple | Applebot-Extended | Applebot-Extended * | yes | - | see entry |
| Google-Extended | Google-Extended | Google-Extended * | yes | - | see entry | |
| FacebookBot | Meta | FacebookBot | FacebookBot | yes | - | see entry |
| Meta-ExternalAgent | Meta | meta-externalagent | meta-externalagent | yes | - | see entry |
| MistralAI-Training | Mistral AI | MistralAI-Training | MistralAI-Training | yes | - | see entry |
| GPTBot | OpenAI | GPTBot | GPTBot | yes | 21 | published ranges |
| Amzn-SearchBot | Amazon | Amzn-SearchBot | Amzn-SearchBot | yes | 513 | published ranges |
| Claude-SearchBot | Anthropic | Claude-SearchBot | Claude-SearchBot | yes | 26 | published ranges |
| DuckAssistBot | DuckDuckGo | DuckAssistBot | DuckAssistBot | yes | 486 | published ranges |
| Google-CloudVertexBot | Google-CloudVertexBot | Google-CloudVertexBot | yes | 317 | reverse DNS (googlebot.com (crawl-***-***-***-***.googlebot.com or geo-crawl-***-***-***-***.geo.googlebot.com)) | |
| Meta-WebIndexer | Meta | meta-webindexer | meta-webindexer | yes | - | see entry |
| MistralAI-Index | Mistral AI | MistralAI-Index | MistralAI-Index | undocumented | 2 | published ranges |
| OAI-SearchBot | OpenAI | OAI-SearchBot | OAI-SearchBot | yes | 35 | published ranges |
| PerplexityBot | Perplexity | PerplexityBot | PerplexityBot | yes | 8 | published ranges |
| YandexAdditionalBot | Yandex | YandexAdditionalBot | YandexAdditionalBot * | partial | - | reverse DNS (yandex.ru, yandex.net, yandex.com) |
| Amzn-User | Amazon | Amzn-User | Amzn-User | partial | 1024 | published ranges |
| Claude-User | Anthropic | Claude-User | Claude-User | yes | 26 | published ranges |
| Chrome Web Store | Google-CWS | undocumented | no | 496 | reverse DNS (gae.googleusercontent.com (***-***-***-***.gae.googleusercontent.com) or google.com (google-proxy-***-***-***-***.google.com)) | |
| Feedfetcher | FeedFetcher-Google | undocumented | no | 496 | reverse DNS (gae.googleusercontent.com (***-***-***-***.gae.googleusercontent.com) or google.com (google-proxy-***-***-***-***.google.com)) | |
| Gemini Notebook | Google-GeminiNotebook | undocumented | no | 496 | reverse DNS (gae.googleusercontent.com (***-***-***-***.gae.googleusercontent.com) or google.com (google-proxy-***-***-***-***.google.com)) | |
| Google Pinpoint | Google-Pinpoint | undocumented | no | 496 | reverse DNS (gae.googleusercontent.com (***-***-***-***.gae.googleusercontent.com) or google.com (google-proxy-***-***-***-***.google.com)) | |
| Google Publisher Center | GoogleProducer | undocumented | no | 496 | reverse DNS (gae.googleusercontent.com (***-***-***-***.gae.googleusercontent.com) or google.com (google-proxy-***-***-***-***.google.com)) | |
| Google Read Aloud | Google-Read-Aloud | undocumented | no | 496 | reverse DNS (gae.googleusercontent.com (***-***-***-***.gae.googleusercontent.com) or google.com (google-proxy-***-***-***-***.google.com)) | |
| Google Site Verifier | Google-Site-Verification | undocumented | no | 496 | reverse DNS (gae.googleusercontent.com (***-***-***-***.gae.googleusercontent.com) or google.com (google-proxy-***-***-***-***.google.com)) | |
| Google-Agent | Google-Agent | undocumented | no | 20 | reverse DNS (undocumented for agents specifically; the user-triggered fetchers page gives ***-***-***-***.gae.googleusercontent.com or google-proxy-***-***-***-***.google.com for the category) | |
| Meta-ExternalFetcher | Meta | meta-externalfetcher | meta-externalfetcher | partial | - | see entry |
| MistralAI-User | Mistral AI | MistralAI-User | MistralAI-User | undocumented | 4 | published ranges |
| ChatGPT-User | OpenAI | ChatGPT-User | undocumented | partial | 207 | published ranges |
| Perplexity-User | Perplexity | Perplexity-User | Perplexity-User | no | 4 | published ranges |
| YandexCalendar | Yandex | YandexCalendar | YandexCalendar | partial | - | reverse DNS (yandex.ru, yandex.net, yandex.com) |
| YandexDialogs | Yandex | YandexDialogs | YandexDialogs | partial | - | reverse DNS (yandex.ru, yandex.net, yandex.com) |
| YandexFeed | Yandex | YandexFeed | YandexFeed | partial | - | reverse DNS (yandex.ru, yandex.net, yandex.com) |
| YandexSearchShop | Yandex | YandexSearchShop | YandexSearchShop | partial | - | reverse DNS (yandex.ru, yandex.net, yandex.com) |
| YandexUserproxy | Yandex | YandexUserproxy | YandexUserproxy | partial | - | reverse DNS (yandex.ru, yandex.net, yandex.com) |
| Applebot | Apple | Applebot | Applebot | yes | 33 | reverse DNS (.applebot.apple.com) |
| Bytespider | ByteDance | Bytespider | undocumented | undocumented | 12 | reverse DNS (.bytedance.com) |
| DuckDuckBot | DuckDuckGo | DuckDuckBot | undocumented | yes | 486 | published ranges |
| Google StoreBot | Storebot-Google | Storebot-Google | yes | 317 | reverse DNS (googlebot.com (crawl-***-***-***-***.googlebot.com or geo-crawl-***-***-***-***.geo.googlebot.com)) | |
| Googlebot | Googlebot | Googlebot | yes | 317 | reverse DNS (googlebot.com (crawl-***-***-***-***.googlebot.com or geo-crawl-***-***-***-***.geo.googlebot.com)) | |
| Googlebot Image | Googlebot-Image | Googlebot-Image | yes | 317 | reverse DNS (googlebot.com (crawl-***-***-***-***.googlebot.com or geo-crawl-***-***-***-***.geo.googlebot.com)) | |
| Googlebot News | Googlebot-News | Googlebot-News * | yes | 317 | reverse DNS (googlebot.com (crawl-***-***-***-***.googlebot.com or geo-crawl-***-***-***-***.geo.googlebot.com)) | |
| Googlebot Video | Googlebot-Video | Googlebot-Video | yes | 317 | reverse DNS (googlebot.com (crawl-***-***-***-***.googlebot.com or geo-crawl-***-***-***-***.geo.googlebot.com)) | |
| BingVideoPreview | Microsoft Bing | BingVideoPreview/1.0 | undocumented | undocumented | - | see entry |
| Bingbot | Microsoft Bing | bingbot/2.0 | Bingbot | yes | 28 | reverse DNS (search.msn.com) |
| YaComBot | Yandex | YaComBot | YaComBot | yes | - | reverse DNS (yandex.ru, yandex.net, yandex.com) |
| YandexBlogs | Yandex | YandexBlogs | YandexBlogs | yes | - | reverse DNS (yandex.ru, yandex.net, yandex.com) |
| YandexBot | Yandex | YandexBot | YandexBot | yes | - | reverse DNS (yandex.ru, yandex.net, yandex.com) |
| YandexComBot | Yandex | YandexComBot | YandexComBot | partial | - | reverse DNS (yandex.ru, yandex.net, yandex.com) |
| YandexFavicons | Yandex | YandexFavicons | YandexFavicons | partial | - | reverse DNS (yandex.ru, yandex.net, yandex.com) |
| YandexImages | Yandex | YandexImages | YandexImages | yes | - | reverse DNS (yandex.ru, yandex.net, yandex.com) |
| YandexMedia | Yandex | YandexMedia | YandexMedia | yes | - | reverse DNS (yandex.ru, yandex.net, yandex.com) |
| YandexMobileBot | Yandex | YandexMobileBot | YandexMobileBot | partial | - | reverse DNS (yandex.ru, yandex.net, yandex.com) |
| YandexOntoDB | Yandex | YandexOntoDB | YandexOntoDB | yes | - | reverse DNS (yandex.ru, yandex.net, yandex.com) |
| YandexRenderResourcesBot | Yandex | YandexRenderResourcesBot | YandexRenderResourcesBot | partial | - | reverse DNS (yandex.ru, yandex.net, yandex.com) |
| YandexSitelinks | Yandex | YandexSitelinks | YandexSitelinks | yes | - | reverse DNS (yandex.ru, yandex.net, yandex.com) |
| YandexVerticals | Yandex | YandexVerticals | YandexVerticals | yes | - | reverse DNS (yandex.ru, yandex.net, yandex.com) |
| YandexVertis | Yandex | YandexVertis | YandexVertis | yes | - | reverse DNS (yandex.ru, yandex.net, yandex.com) |
| YandexVideo | Yandex | YandexVideo | YandexVideo | yes | - | reverse DNS (yandex.ru, yandex.net, yandex.com) |
| YandexVideoParser | Yandex | YandexVideoParser | YandexVideoParser | partial | - | reverse DNS (yandex.ru, yandex.net, yandex.com) |
| Google Messages | GoogleMessages | undocumented | no | 496 | reverse DNS (gae.googleusercontent.com (***-***-***-***.gae.googleusercontent.com) or google.com (google-proxy-***-***-***-***.google.com)) | |
| FacebookExternalHit | Meta | facebookexternalhit | facebookexternalhit | partial | - | see entry |
| MicrosoftPreview | Microsoft Bing | MicrosoftPreview/2.0 | undocumented | undocumented | - | see entry |
| AhrefsBot | Ahrefs Pte. Ltd. | AhrefsBot | AhrefsBot | yes | 81 | reverse DNS (ahrefs.com or ahrefs.net) |
| AhrefsSiteAudit | Ahrefs Pte. Ltd. | AhrefsSiteAudit | AhrefsSiteAudit | partial | 81 | reverse DNS (ahrefs.com or ahrefs.net) |
| Google-InspectionTool | Google-InspectionTool | Google-InspectionTool | yes | 317 | reverse DNS (googlebot.com (crawl-***-***-***-***.googlebot.com or geo-crawl-***-***-***-***.geo.googlebot.com)) | |
| RyteBot | Semrush | RyteBot | RyteBot | yes | - | see entry |
| SemrushBot | Semrush | SemrushBot | SemrushBot | yes | - | see entry |
| SemrushBot-BA | Semrush | SemrushBot-BA | SemrushBot-BA | yes | - | see entry |
| SemrushBot-ESI | Semrush | SemrushBot-ESI | SemrushBot-ESI | yes | - | see entry |
| SemrushBot-FT | Semrush | SemrushBot-FT | SemrushBot-FT | yes | - | see entry |
| SemrushBot-OCOB | Semrush | SemrushBot-OCOB | SemrushBot-OCOB | yes | - | see entry |
| SemrushBot-SI | Semrush | SemrushBot-SI | SemrushBot-SI | yes | - | see entry |
| SemrushBot-SWA | Semrush | SemrushBot-SWA | SemrushBot-SWA | yes | - | see entry |
| SiteAuditBot | Semrush | SiteAuditBot | SiteAuditBot | partial | - | see entry |
| SplitSignalBot | Semrush | SplitSignalBot | SplitSignalBot | yes | - | see entry |
| YandexPagechecker | Yandex | YandexPagechecker | YandexPagechecker | yes | - | reverse DNS (yandex.ru, yandex.net, yandex.com) |
| YandexWebmaster | Yandex | YandexWebmaster | YandexWebmaster | yes | - | reverse DNS (yandex.ru, yandex.net, yandex.com) |
| CCBot | Common Crawl | CCBot | CCBot | yes | 5 | reverse DNS (.crawl.commoncrawl.org) |
| iTMS | Apple | iTMS | none (does not follow robots.txt) | no | - | reverse DNS (.applebot.apple.com) |
| APIs-Google | APIs-Google | APIs-Google | partial | 272 | reverse DNS (google.com (rate-limited-proxy-***-***-***-***.google.com)) | |
| AdSense | Mediapartners-Google | Mediapartners-Google | partial | 272 | reverse DNS (google.com (rate-limited-proxy-***-***-***-***.google.com)) | |
| AdsBot | AdsBot-Google | AdsBot-Google | partial | 272 | reverse DNS (google.com (rate-limited-proxy-***-***-***-***.google.com)) | |
| AdsBot Mobile Web | AdsBot-Google-Mobile | AdsBot-Google-Mobile | partial | 272 | reverse DNS (google.com (rate-limited-proxy-***-***-***-***.google.com)) | |
| Google-Safety | Google-Safety | undocumented | no | 272 | reverse DNS (google.com (rate-limited-proxy-***-***-***-***.google.com)) | |
| GoogleOther | GoogleOther | GoogleOther | yes | 317 | reverse DNS (googlebot.com (crawl-***-***-***-***.googlebot.com or geo-crawl-***-***-***-***.geo.googlebot.com)) | |
| GoogleOther-Image | GoogleOther-Image | GoogleOther-Image | yes | 317 | reverse DNS (googlebot.com (crawl-***-***-***-***.googlebot.com or geo-crawl-***-***-***-***.geo.googlebot.com)) | |
| GoogleOther-Video | GoogleOther-Video | GoogleOther-Video | yes | 317 | reverse DNS (googlebot.com (crawl-***-***-***-***.googlebot.com or geo-crawl-***-***-***-***.geo.googlebot.com)) | |
| Meta-ExternalAds | Meta | meta-externalads | meta-externalads | yes | - | see entry |
| AdIdxBot | Microsoft Bing | adidxbot/2.0 | AdIdxbot | undocumented | - | see entry |
| OAI-AdsBot | OpenAI | OAI-AdsBot | undocumented | undocumented | 2 | published ranges |
| YaDirectFetcher | Yandex | YaDirectFetcher | YaDirectFetcher | no | - | reverse DNS (yandex.ru, yandex.net, yandex.com) |
| Yandex | Yandex | Yandex | Yandex * | partial | - | see entry |
| YandexAccessibilityBot | Yandex | YandexAccessibilityBot | YandexAccessibilityBot | partial | - | reverse DNS (yandex.ru, yandex.net, yandex.com) |
| YandexAdNet | Yandex | YandexAdNet | YandexAdNet | yes | - | reverse DNS (yandex.ru, yandex.net, yandex.com) |
| YandexCheckBot | Yandex | YandexCheckBot | YandexCheckBot | partial | - | reverse DNS (yandex.ru, yandex.net, yandex.com) |
| YandexDirect | Yandex | YandexDirect | YandexDirect | partial | - | reverse DNS (yandex.ru, yandex.net, yandex.com) |
| YandexDirectDyn | Yandex | YandexDirectDyn | YandexDirectDyn | partial | - | reverse DNS (yandex.ru, yandex.net, yandex.com) |
| YandexImageResizer | Yandex | YandexImageResizer | YandexImageResizer | yes | - | reverse DNS (yandex.ru, yandex.net, yandex.com) |
| YandexMarket | Yandex | YandexMarket | YandexMarket | partial | - | reverse DNS (yandex.ru, yandex.net, yandex.com) |
| YandexMetrika | Yandex | YandexMetrika | YandexMetrika | no | - | reverse DNS (yandex.ru, yandex.net, yandex.com) |
| YandexMobileScreenShotBot | Yandex | YandexMobileScreenShotBot | YandexMobileScreenShotBot | partial | - | reverse DNS (yandex.ru, yandex.net, yandex.com) |
| YandexOntoDBAPI | Yandex | YandexOntoDBAPI | YandexOntoDBAPI | partial | - | reverse DNS (yandex.ru, yandex.net, yandex.com) |
| YandexPartner | Yandex | YandexPartner | YandexPartner | partial | - | reverse DNS (yandex.ru, yandex.net, yandex.com) |
| YandexRCA | Yandex | YandexRCA | YandexRCA | partial | - | reverse DNS (yandex.ru, yandex.net, yandex.com) |
| YandexScreenshotBot | Yandex | YandexScreenshotBot | YandexScreenshotBot | partial | - | reverse DNS (yandex.ru, yandex.net, yandex.com) |
| YandexSpravBot | Yandex | YandexSpravBot | YandexSpravBot | yes | - | reverse DNS (yandex.ru, yandex.net, yandex.com) |
| YandexTracker | Yandex | YandexTracker | YandexTracker | partial | - | reverse DNS (yandex.ru, yandex.net, yandex.com) |
* policy token with no crawler of its own. IP prefix counts were read from each operator's official file on 2026-09-06.
Method
Inclusion requires official documentation from the operator. One reviewer compiles each entry from those pages; a second reviewer re-opens every cited URL and removes any claim that is not on it. Fields the operator does not document are recorded as undocumented rather than inferred.
IP range figures are produced by scripts/bot-registry/fetch-ranges.mjs, which downloads each official range file, counts IPv4 and IPv6 prefixes, and stores the file's stated creation time and a SHA-256 of the body. Third-party IP lists are never used.
Reviewed September 2026. Ranges fetched 2026-09-06.
Cite this dataset
APA
PROXIES.SX. (2026). Crawler & Bot Registry: 109 documented web crawlers from 15 operators, with user agents, robots.txt tokens and official IP range files. Reviewed September 2026. https://www.proxies.sx/crawlers/dataset
BibTeX
@misc{proxiessx_crawler_registry,
title = {Crawler & Bot Registry: 109 documented web crawlers from 15 operators},
author = {{PROXIES.SX}},
year = {2026},
note = {Reviewed September 2026. CC BY 4.0},
url = {https://www.proxies.sx/crawlers/dataset}
}More open data
The Anti-Bot Census documents the protection popular sites run against automated visitors. Both datasets live on the research hub.
Research & open datasets