AI training crawlers
Collects public web content that may be used to train or improve AI models. These are the tokens most site owners decide about first.
AI search indexs
Builds an index that an AI assistant searches at answer time, so it can cite and link pages. Blocking it removes a site from those answers.
User-triggered fetchers
Fetches a page because a person asked an assistant or app to open it. Operators generally treat these like a browser acting on the user's behalf.
Search engine crawlers
Classic web search indexing. Blocking it removes a site from that search engine.
SEO tool crawlers
Crawls to build backlink and site-audit indexes sold to marketers.
Archive crawlers
Crawls to build a public archive or open corpus of the web.
Others
Documented bots that do not fit the categories above.
By operator
- YandexAdditionalBot
- YandexCalendar
- YandexDialogs
- YandexFeed
- YandexSearchShop
- YandexUserproxy
- YaComBot
- YandexBlogs
- YandexBot
- YandexComBot
- YandexFavicons
- YandexImages
- YandexMedia
- YandexMobileBot
- YandexOntoDB
- YandexRenderResourcesBot
- YandexSitelinks
- YandexVerticals
- YandexVertis
- YandexVideo
- YandexVideoParser
- YandexPagechecker
- YandexWebmaster
- YaDirectFetcher
- Yandex
- YandexAccessibilityBot
- YandexAdNet
- YandexCheckBot
- YandexDirect
- YandexDirectDyn
- YandexImageResizer
- YandexMarket
- YandexMetrika
- YandexMobileScreenShotBot
- YandexOntoDBAPI
- YandexPartner
- YandexRCA
- YandexScreenshotBot
- YandexSpravBot
- YandexTracker
- Google-Extended
- Google-CloudVertexBot
- Chrome Web Store
- Feedfetcher
- Gemini Notebook
- Google Pinpoint
- Google Publisher Center
- Google Read Aloud
- Google Site Verifier
- Google-Agent
- Google StoreBot
- Googlebot
- Googlebot Image
- Googlebot News
- Googlebot Video
- Google Messages
- Google-InspectionTool
- APIs-Google
- AdSense
- AdsBot
- AdsBot Mobile Web
- Google-Safety
- GoogleOther
- GoogleOther-Image
- GoogleOther-Video
How this registry is built
Each entry was compiled from the operator's own documentation, then handed to a second reviewer whose only job was to re-open every source and strike any claim not found there. What survived is what you see. Fields the operator does not document are marked as such rather than filled in.
IP-range figures come from a script that downloads each operator's official range file, counts the prefixes, and records the file's stated creation time and a SHA-256 of the body. The last run was 2026-09-06. Every bot page links the live file so you can compare.
The registry is read-only reference. It documents what operators say about their own bots so site owners can identify and control them; it does not publish third-party IP lists or speculate about undocumented behaviour.
Common questions
What is the crawler and bot registry?+
A read-only reference of 109 web crawlers and fetchers from 15 operators, including the AI crawlers site owners ask about most: GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot and Googlebot. Each entry gives the exact user-agent token, the robots.txt token, whether the operator says it honors robots.txt, the documented way to verify a request, and the official IP range file where one is published. Reviewed September 2026.
Where does the data come from?+
Only from each operator's own documentation and official machine-readable range files. Every field was checked against the operator's page and re-checked by an independent reviewer. IP-range counts are produced by a script that downloads the operator's file and records its creation time and a hash, last run 2026-09-06. Bots without official documentation are not listed.
How do I block AI crawlers from training on my site?+
Add a robots.txt rule per token. Training crawlers documented here include GPTBot, ClaudeBot and CCBot; Google uses the policy token Google-Extended, which has no crawler of its own. Each entry has copy-ready allow and block blocks. Note that user-triggered fetchers such as ChatGPT-User act on a person's request and some operators document that they do not consult robots.txt.
How can I tell whether a request is really from the bot it claims to be?+
Anyone can send any User-Agent string. The two documented checks are a reverse DNS lookup (the IP must resolve to the operator's domain and back again) and a match against the operator's published IP ranges. Each entry states which of those the operator supports.
Is the dataset free to use?+
Yes. The full registry is published as an open dataset under CC BY 4.0, downloadable as CSV, NDJSON or JSON, with citation blocks on the dataset page.
Related references
The anti-bot protection registry covers the other side of this exchange: which protection popular sites run against automated visitors. The glossary defines the terms both registries use.
All open datasets
Social preview fetchers
Fetches a page to build the link card shown when a URL is shared on a platform.