Yandex
Yandex is not a crawler but the group token that Yandex bots share in robots.txt. Yandex's indexing bot matches the substring Yandex, case-insensitive, in User-agent records; every Yandex bot without its own block follows User-agent: Yandex, and when that block exists User-agent: * is ignored. There is no verification method, because it is a token rather than a crawler.
What Yandex does
The entry has no user-agent string, because Yandex is a policy token rather than a fetching agent. Its role is inside robots.txt. Yandex's official example explains the hierarchy. User-agent: YandexBot is read only by the main indexing bot, User-agent: Yandex by all Yandex bots except those with their own block, and User-agent: * is not used by Yandex bots when a Yandex block exists. A bot with its own more specific block follows that block instead of the shared one, so a rule written for the group token can be overridden by a per-bot block placed elsewhere in the file.
Compliance is partial. Yandex warns that some of its robots may ignore directives even under User-agent: Yandex and must be addressed by their exact User-agent name. So a Disallow under the group token blocks most Yandex bots but not necessarily all of them; check each bot's own entry for exceptions. Yandex has ignored crawl-delay since 22 February 2018; crawl rate is set in Yandex Webmaster instead. Because this is a token and not a crawler, there is no verification method for it. Verify the individual Yandex crawlers through their own registry entries.
Operator note. This is a group robots.txt token rather than a crawler. In Yandex's official example, 'User-agent: YandexBot' is used only by the main indexing bot, 'User-agent: Yandex' by all Yandex bots except those with their own block, and 'User-agent: *' is not used by Yandex bots when a Yandex block exists. Yandex warns that some of its robots may ignore directives even under 'User-agent: Yandex' and must be addressed by their exact User-agent. Crawl-delay has been ignored since 22 Feb 2018; crawl rate is set in Yandex Webmaster instead.
Controlling Yandex with robots.txt
Yandex is a policy token. It has no crawler of its own; it tells Yandex how content already fetched by its other crawlers may be used. Put it in robots.txt like any other token.
User-agent: Yandex
Disallow: /User-agent: Yandex
Allow: /Verifying a request is really Yandex
Anyone can put Yandex in a User-Agent header. The operator does not document a verification method.
Common questions
Should I block Yandex in robots.txt?
A Disallow under User-agent: Yandex applies to all Yandex bots that lack their own block, so it is the broadest single rule for Yandex traffic. Yandex warns that some robots may ignore it and need their exact User-agent named. The group rule covers the main indexing bot unless YandexBot has its own block.
Is Yandex a crawler?
No. Yandex is a group token for robots.txt. Yandex bots look for the substring Yandex, case-insensitive, in User-agent records, and any bot without a more specific block follows those rules. The entry has no user-agent string; named bots such as YandexBot have their own entries.
Does Yandex honor crawl-delay?
No. Yandex documents that it has ignored crawl-delay since 22 February 2018. Crawl rate for Yandex bots is set in Yandex Webmaster instead of robots.txt, so a crawl-delay line under User-agent: Yandex has no effect.
Sources
Every fact on this page was checked against Yandex's own documentation, listed below, and re-checked by a second reviewer before publication. Reviewed September 2026.
Related crawlers
This registry documents how operators describe their own bots so site owners can identify and control them. It does not publish third-party IP lists or guess at undocumented behaviour. To see how your own site responds to automated visitors, the bot detection scanner reads a URL's live response and names the protection it finds.