Open datasetCC BY 4.0 · reviewed September 2026
Crawler & Bot Registry dataset
109 documented crawlers and fetchers from 15 operators, one row each: user-agent token, robots.txt token, whether the operator says it honors robots.txt, the official IP range file with its prefix count and creation time, the documented verification method, and the source documentation URL. Free to use with attribution.
109
documented bots
46
with official IP range files
12,153
IP prefixes across those files
55/109
documented as honoring robots.txt
By purpose
Every row
| Bot | Operator | User-agent token | robots.txt token | Honors robots | IP prefixes | Verification |
|---|---|---|---|---|---|---|
| Amazonbot | Amazon | Amazonbot | Amazonbot | yes | 525 | published ranges |
| ClaudeBot | Anthropic | ClaudeBot | ClaudeBot | yes | 26 | published ranges |
| Applebot-Extended | Apple | Applebot-Extended | Applebot-Extended * | yes | - | see entry |
| Google-Extended | Google-Extended | Google-Extended * | yes | - | see entry | |
| FacebookBot | Meta | FacebookBot | FacebookBot | yes | - | see entry |
| Meta-ExternalAgent | Meta | meta-externalagent | meta-externalagent | yes | - | see entry |
| MistralAI-Training | Mistral AI | MistralAI-Training | MistralAI-Training | yes | - | see entry |
| GPTBot | OpenAI | GPTBot | GPTBot | yes | 21 | published ranges |
| Amzn-SearchBot | Amazon | Amzn-SearchBot | Amzn-SearchBot | yes | 513 | published ranges |
| Claude-SearchBot | Anthropic | Claude-SearchBot | Claude-SearchBot | yes | 26 | published ranges |
| DuckAssistBot | DuckDuckGo | DuckAssistBot | DuckAssistBot | yes | 486 | published ranges |
| Google-CloudVertexBot | Google-CloudVertexBot | Google-CloudVertexBot | yes | 317 | reverse DNS (googlebot.com (crawl-***-***-***-***.googlebot.com or geo-crawl-***-***-***-***.geo.googlebot.com)) | |
| Meta-WebIndexer | Meta | meta-webindexer | meta-webindexer | yes | - | see entry |
| MistralAI-Index | Mistral AI | MistralAI-Index | MistralAI-Index | undocumented | 2 | published ranges |
| OAI-SearchBot | OpenAI | OAI-SearchBot | OAI-SearchBot | yes | 35 | published ranges |
| PerplexityBot | Perplexity | PerplexityBot | PerplexityBot | yes | 8 | published ranges |
| YandexAdditionalBot | Yandex | YandexAdditionalBot | YandexAdditionalBot * | partial | - | reverse DNS (yandex.ru, yandex.net, yandex.com) |
| Amzn-User | Amazon | Amzn-User | Amzn-User | partial | 1024 | published ranges |
| Claude-User | Anthropic | Claude-User | Claude-User | yes | 26 | published ranges |
| Chrome Web Store | Google-CWS | undocumented | no | 496 | reverse DNS (gae.googleusercontent.com (***-***-***-***.gae.googleusercontent.com) or google.com (google-proxy-***-***-***-***.google.com)) | |
| Feedfetcher | FeedFetcher-Google | undocumented | no | 496 | reverse DNS (gae.googleusercontent.com (***-***-***-***.gae.googleusercontent.com) or google.com (google-proxy-***-***-***-***.google.com)) | |
| Gemini Notebook | Google-GeminiNotebook | undocumented | no | 496 | reverse DNS (gae.googleusercontent.com (***-***-***-***.gae.googleusercontent.com) or google.com (google-proxy-***-***-***-***.google.com)) | |
| Google Pinpoint | Google-Pinpoint | undocumented | no | 496 | reverse DNS (gae.googleusercontent.com (***-***-***-***.gae.googleusercontent.com) or google.com (google-proxy-***-***-***-***.google.com)) | |
| Google Publisher Center | GoogleProducer | undocumented | no | 496 | reverse DNS (gae.googleusercontent.com (***-***-***-***.gae.googleusercontent.com) or google.com (google-proxy-***-***-***-***.google.com)) | |
| Google Read Aloud | Google-Read-Aloud | undocumented | no | 496 | reverse DNS (gae.googleusercontent.com (***-***-***-***.gae.googleusercontent.com) or google.com (google-proxy-***-***-***-***.google.com)) | |
| Google Site Verifier | Google-Site-Verification | undocumented | no | 496 | reverse DNS (gae.googleusercontent.com (***-***-***-***.gae.googleusercontent.com) or google.com (google-proxy-***-***-***-***.google.com)) | |
| Google-Agent | Google-Agent | undocumented | no | 20 | reverse DNS (undocumented for agents specifically; the user-triggered fetchers page gives ***-***-***-***.gae.googleusercontent.com or google-proxy-***-***-***-***.google.com for the category) | |
| Meta-ExternalFetcher | Meta | meta-externalfetcher | meta-externalfetcher | partial | - | see entry |
| MistralAI-User | Mistral AI | MistralAI-User | MistralAI-User | undocumented | 4 | published ranges |
| ChatGPT-User | OpenAI | ChatGPT-User | undocumented | partial | 207 | published ranges |
| Perplexity-User | Perplexity | Perplexity-User | Perplexity-User | no | 4 | published ranges |
| YandexCalendar | Yandex | YandexCalendar | YandexCalendar | partial | - | reverse DNS (yandex.ru, yandex.net, yandex.com) |
| YandexDialogs | Yandex | YandexDialogs | YandexDialogs | partial | - | reverse DNS (yandex.ru, yandex.net, yandex.com) |
| YandexFeed | Yandex | YandexFeed | YandexFeed | partial | - | reverse DNS (yandex.ru, yandex.net, yandex.com) |
| YandexSearchShop | Yandex | YandexSearchShop | YandexSearchShop | partial | - | reverse DNS (yandex.ru, yandex.net, yandex.com) |
| YandexUserproxy | Yandex | YandexUserproxy | YandexUserproxy | partial | - | reverse DNS (yandex.ru, yandex.net, yandex.com) |
| Applebot | Apple | Applebot | Applebot | yes | 33 | reverse DNS (.applebot.apple.com) |
| Bytespider | ByteDance | Bytespider | undocumented | undocumented | 12 | reverse DNS (.bytedance.com) |
| DuckDuckBot | DuckDuckGo | DuckDuckBot | undocumented | yes | 486 | published ranges |
| Google StoreBot | Storebot-Google | Storebot-Google | yes | 317 | reverse DNS (googlebot.com (crawl-***-***-***-***.googlebot.com or geo-crawl-***-***-***-***.geo.googlebot.com)) | |
| Googlebot | Googlebot | Googlebot | yes | 317 | reverse DNS (googlebot.com (crawl-***-***-***-***.googlebot.com or geo-crawl-***-***-***-***.geo.googlebot.com)) | |
| Googlebot Image | Googlebot-Image | Googlebot-Image | yes | 317 | reverse DNS (googlebot.com (crawl-***-***-***-***.googlebot.com or geo-crawl-***-***-***-***.geo.googlebot.com)) | |
| Googlebot News | Googlebot-News | Googlebot-News * | yes | 317 | reverse DNS (googlebot.com (crawl-***-***-***-***.googlebot.com or geo-crawl-***-***-***-***.geo.googlebot.com)) | |
| Googlebot Video | Googlebot-Video | Googlebot-Video | yes | 317 | reverse DNS (googlebot.com (crawl-***-***-***-***.googlebot.com or geo-crawl-***-***-***-***.geo.googlebot.com)) | |
| BingVideoPreview | Microsoft Bing | BingVideoPreview/1.0 | undocumented | undocumented | - | see entry |
| Bingbot | Microsoft Bing | bingbot/2.0 | Bingbot | yes | 28 | reverse DNS (search.msn.com) |
| YaComBot | Yandex | YaComBot | YaComBot | yes | - | reverse DNS (yandex.ru, yandex.net, yandex.com) |
| YandexBlogs | Yandex | YandexBlogs | YandexBlogs | yes | - | reverse DNS (yandex.ru, yandex.net, yandex.com) |
| YandexBot | Yandex | YandexBot | YandexBot | yes | - | reverse DNS (yandex.ru, yandex.net, yandex.com) |
| YandexComBot | Yandex | YandexComBot | YandexComBot | partial | - | reverse DNS (yandex.ru, yandex.net, yandex.com) |
| YandexFavicons | Yandex | YandexFavicons | YandexFavicons | partial | - | reverse DNS (yandex.ru, yandex.net, yandex.com) |
| YandexImages | Yandex | YandexImages | YandexImages | yes | - | reverse DNS (yandex.ru, yandex.net, yandex.com) |
| YandexMedia | Yandex | YandexMedia | YandexMedia | yes | - | reverse DNS (yandex.ru, yandex.net, yandex.com) |
| YandexMobileBot | Yandex | YandexMobileBot | YandexMobileBot | partial | - | reverse DNS (yandex.ru, yandex.net, yandex.com) |
| YandexOntoDB | Yandex | YandexOntoDB | YandexOntoDB | yes | - | reverse DNS (yandex.ru, yandex.net, yandex.com) |
| YandexRenderResourcesBot | Yandex | YandexRenderResourcesBot | YandexRenderResourcesBot | partial | - | reverse DNS (yandex.ru, yandex.net, yandex.com) |
| YandexSitelinks | Yandex | YandexSitelinks | YandexSitelinks | yes | - | reverse DNS (yandex.ru, yandex.net, yandex.com) |
| YandexVerticals | Yandex | YandexVerticals | YandexVerticals | yes | - | reverse DNS (yandex.ru, yandex.net, yandex.com) |
| YandexVertis | Yandex | YandexVertis | YandexVertis | yes | - | reverse DNS (yandex.ru, yandex.net, yandex.com) |
| YandexVideo | Yandex | YandexVideo | YandexVideo | yes | - | reverse DNS (yandex.ru, yandex.net, yandex.com) |
| YandexVideoParser | Yandex | YandexVideoParser | YandexVideoParser | partial | - | reverse DNS (yandex.ru, yandex.net, yandex.com) |
| Google Messages | GoogleMessages | undocumented | no | 496 | reverse DNS (gae.googleusercontent.com (***-***-***-***.gae.googleusercontent.com) or google.com (google-proxy-***-***-***-***.google.com)) | |
| FacebookExternalHit | Meta | facebookexternalhit | facebookexternalhit | partial | - | see entry |
| MicrosoftPreview | Microsoft Bing | MicrosoftPreview/2.0 | undocumented | undocumented | - | see entry |
| AhrefsBot | Ahrefs Pte. Ltd. | AhrefsBot | AhrefsBot | yes | 81 | reverse DNS (ahrefs.com or ahrefs.net) |
| AhrefsSiteAudit | Ahrefs Pte. Ltd. | AhrefsSiteAudit | AhrefsSiteAudit | partial | 81 | reverse DNS (ahrefs.com or ahrefs.net) |
| Google-InspectionTool | Google-InspectionTool | Google-InspectionTool | yes | 317 | reverse DNS (googlebot.com (crawl-***-***-***-***.googlebot.com or geo-crawl-***-***-***-***.geo.googlebot.com)) | |
| RyteBot | Semrush | RyteBot | RyteBot | yes | - | see entry |
| SemrushBot | Semrush | SemrushBot | SemrushBot | yes | - | see entry |
| SemrushBot-BA | Semrush | SemrushBot-BA | SemrushBot-BA | yes | - | see entry |
| SemrushBot-ESI | Semrush | SemrushBot-ESI | SemrushBot-ESI | yes | - | see entry |
| SemrushBot-FT | Semrush | SemrushBot-FT | SemrushBot-FT | yes | - | see entry |
| SemrushBot-OCOB | Semrush | SemrushBot-OCOB | SemrushBot-OCOB | yes | - | see entry |
| SemrushBot-SI | Semrush | SemrushBot-SI | SemrushBot-SI | yes | - | see entry |
| SemrushBot-SWA | Semrush | SemrushBot-SWA | SemrushBot-SWA | yes | - | see entry |
| SiteAuditBot | Semrush | SiteAuditBot | SiteAuditBot | partial | - | see entry |
| SplitSignalBot | Semrush | SplitSignalBot | SplitSignalBot | yes | - | see entry |
| YandexPagechecker | Yandex | YandexPagechecker | YandexPagechecker | yes | - | reverse DNS (yandex.ru, yandex.net, yandex.com) |
| YandexWebmaster | Yandex | YandexWebmaster | YandexWebmaster | yes | - | reverse DNS (yandex.ru, yandex.net, yandex.com) |
| CCBot | Common Crawl | CCBot | CCBot | yes | 5 | reverse DNS (.crawl.commoncrawl.org) |
| iTMS | Apple | iTMS | none (does not follow robots.txt) | no | - | reverse DNS (.applebot.apple.com) |
| APIs-Google | APIs-Google | APIs-Google | partial | 272 | reverse DNS (google.com (rate-limited-proxy-***-***-***-***.google.com)) | |
| AdSense | Mediapartners-Google | Mediapartners-Google | partial | 272 | reverse DNS (google.com (rate-limited-proxy-***-***-***-***.google.com)) | |
| AdsBot | AdsBot-Google | AdsBot-Google | partial | 272 | reverse DNS (google.com (rate-limited-proxy-***-***-***-***.google.com)) | |
| AdsBot Mobile Web | AdsBot-Google-Mobile | AdsBot-Google-Mobile | partial | 272 | reverse DNS (google.com (rate-limited-proxy-***-***-***-***.google.com)) | |
| Google-Safety | Google-Safety | undocumented | no | 272 | reverse DNS (google.com (rate-limited-proxy-***-***-***-***.google.com)) | |
| GoogleOther | GoogleOther | GoogleOther | yes | 317 | reverse DNS (googlebot.com (crawl-***-***-***-***.googlebot.com or geo-crawl-***-***-***-***.geo.googlebot.com)) | |
| GoogleOther-Image | GoogleOther-Image | GoogleOther-Image | yes | 317 | reverse DNS (googlebot.com (crawl-***-***-***-***.googlebot.com or geo-crawl-***-***-***-***.geo.googlebot.com)) | |
| GoogleOther-Video | GoogleOther-Video | GoogleOther-Video | yes | 317 | reverse DNS (googlebot.com (crawl-***-***-***-***.googlebot.com or geo-crawl-***-***-***-***.geo.googlebot.com)) | |
| Meta-ExternalAds | Meta | meta-externalads | meta-externalads | yes | - | see entry |
| AdIdxBot | Microsoft Bing | adidxbot/2.0 | AdIdxbot | undocumented | - | see entry |
| OAI-AdsBot | OpenAI | OAI-AdsBot | undocumented | undocumented | 2 | published ranges |
| YaDirectFetcher | Yandex | YaDirectFetcher | YaDirectFetcher | no | - | reverse DNS (yandex.ru, yandex.net, yandex.com) |
| Yandex | Yandex | Yandex | Yandex * | partial | - | see entry |
| YandexAccessibilityBot | Yandex | YandexAccessibilityBot | YandexAccessibilityBot | partial | - | reverse DNS (yandex.ru, yandex.net, yandex.com) |
| YandexAdNet | Yandex | YandexAdNet | YandexAdNet | yes | - | reverse DNS (yandex.ru, yandex.net, yandex.com) |
| YandexCheckBot | Yandex | YandexCheckBot | YandexCheckBot | partial | - | reverse DNS (yandex.ru, yandex.net, yandex.com) |
| YandexDirect | Yandex | YandexDirect | YandexDirect | partial | - | reverse DNS (yandex.ru, yandex.net, yandex.com) |
| YandexDirectDyn | Yandex | YandexDirectDyn | YandexDirectDyn | partial | - | reverse DNS (yandex.ru, yandex.net, yandex.com) |
| YandexImageResizer | Yandex | YandexImageResizer | YandexImageResizer | yes | - | reverse DNS (yandex.ru, yandex.net, yandex.com) |
| YandexMarket | Yandex | YandexMarket | YandexMarket | partial | - | reverse DNS (yandex.ru, yandex.net, yandex.com) |
| YandexMetrika | Yandex | YandexMetrika | YandexMetrika | no | - | reverse DNS (yandex.ru, yandex.net, yandex.com) |
| YandexMobileScreenShotBot | Yandex | YandexMobileScreenShotBot | YandexMobileScreenShotBot | partial | - | reverse DNS (yandex.ru, yandex.net, yandex.com) |
| YandexOntoDBAPI | Yandex | YandexOntoDBAPI | YandexOntoDBAPI | partial | - | reverse DNS (yandex.ru, yandex.net, yandex.com) |
| YandexPartner | Yandex | YandexPartner | YandexPartner | partial | - | reverse DNS (yandex.ru, yandex.net, yandex.com) |
| YandexRCA | Yandex | YandexRCA | YandexRCA | partial | - | reverse DNS (yandex.ru, yandex.net, yandex.com) |
| YandexScreenshotBot | Yandex | YandexScreenshotBot | YandexScreenshotBot | partial | - | reverse DNS (yandex.ru, yandex.net, yandex.com) |
| YandexSpravBot | Yandex | YandexSpravBot | YandexSpravBot | yes | - | reverse DNS (yandex.ru, yandex.net, yandex.com) |
| YandexTracker | Yandex | YandexTracker | YandexTracker | partial | - | reverse DNS (yandex.ru, yandex.net, yandex.com) |
* policy token with no crawler of its own. IP prefix counts were read from each operator's official file on 2026-09-06.
Questions
How was this dataset collected?
Each bot was researched from its operator's own published documentation, then checked a second time by a separate reviewer. Where an operator publishes a machine-readable IP range file, a script downloads it, counts the prefixes and records the file's creation time. Bots without official documentation are not included.
Can I use it commercially?
Yes. The dataset is published under Creative Commons Attribution 4.0, which allows commercial use, redistribution and derivative works as long as you credit PROXIES.SX and link back to this page. The citation blocks below give a ready-made attribution in APA and BibTeX form.
How often is it updated?
The registry was last reviewed September 2026 and the IP range probe last ran on 2026-09-06. Operators change user-agent tokens and range files without notice, so treat the prefix counts as a snapshot and re-fetch the operator's own file when the exact set matters.
What does each column mean?
The CSV has 15 columns: the bot's slug, name and operator, its exact user-agent token, the purpose category, the robots.txt token and whether that token is policy-only, whether the operator documents that it honors robots.txt, the official IP range URL with its prefix count and creation time, the reverse-DNS suffix, the documented verification method, and the source documentation URL.
Method
Inclusion requires official documentation from the operator. One reviewer compiles each entry from those pages; a second reviewer re-opens every cited URL and removes any claim that is not on it. Fields the operator does not document are recorded as undocumented rather than inferred.
IP range figures are produced by scripts/bot-registry/fetch-ranges.mjs, which downloads each official range file, counts IPv4 and IPv6 prefixes, and stores the file's stated creation time and a SHA-256 of the body. Third-party IP lists are never used.
Reviewed September 2026. Ranges fetched 2026-09-06.
Cite this dataset
APA
PROXIES.SX. (2026). Crawler & Bot Registry: 109 documented web crawlers from 15 operators, with user agents, robots.txt tokens and official IP range files. Reviewed September 2026. https://www.proxies.sx/crawlers/dataset
BibTeX
@misc{proxiessx_crawler_registry,
title = {Crawler & Bot Registry: 109 documented web crawlers from 15 operators},
author = {{PROXIES.SX}},
year = {2026},
note = {Reviewed September 2026. CC BY 4.0},
url = {https://www.proxies.sx/crawlers/dataset}
}More open data
The Anti-Bot Census documents the protection popular sites run against automated visitors. Both datasets live on the research hub.
Research & open datasets