SemrushBot
Semrush operates SemrushBot, an SEO-tool crawler that discovers and collects new and updated web data for its Backlink Analytics webgraph of links; Link Building and Topic Research reports also use the data. It honors robots.txt under the SemrushBot token. Semrush documents no verification method and no IP range file, and asks site owners not to block by IP.
What SemrushBot does
SemrushBot starts from a list of URLs, and when it visits each page it saves the hyperlinks it finds for further crawling. Semrush publishes no full user-agent example on its bot page, so logs will show the SemrushBot token inside whatever string the crawler sends. The robots.txt block for this token is described as stopping crawling for the webgraph of links, the public backlink index behind Backlink Analytics. Semrush lists Link Building and Topic Research reports among other uses of the same data without assigning them a separate token.
Blocking SemrushBot in robots.txt removes a site's link data from Backlink Analytics and the other reports Semrush names, so Semrush's reports on that site lose the crawled link data. Semrush asks site owners not to block by IP because it does not use consecutive IP blocks, and it publishes no verification method, so robots.txt is the documented control. Changes to robots.txt may take up to one hour or 100 requests to be discovered.
Semrush documents specific robots.txt handling. The file must sit at the top directory of the host and on each subdomain, and must return HTTP 200 or a 3xx redirect; a 4xx response is treated as no restrictions, while a 5xx response stops SemrushBot from crawling the whole site. The crawler supports Crawl-delay up to 10 seconds, with higher values cut to that limit, and wildcards. Without a Crawl-delay it adjusts request frequency to current server load. Site Audit contexts allow a delay up to 30. Contact is bot@semrush.com.
Operator note. The crawl "starts with a list of webpage URLs. When SemrushBot visits these URLs, it saves hyperlinks from the page for further crawling", and Semrush states that "SemrushBot for Backlink Analytics also supports the following non-standard extensions to robots.txt: Crawl-delay directives. Our crawler can take intervals of up to 10 seconds between requests to a site. Higher values will be cut down to this 10-second limit. If no crawl-delay is specified, SemrushBot will adjust the frequency of requests to your site according to the current server load. The use of wildcards (*)." The robots.txt file must sit in the top directory of the host and on each subdomain and must return HTTP 200: "If a 4xx status code is returned, SemrushBot will assume that no robots.txt exists and there are no crawl restrictions. Returning a 5xx status code for your robots.txt file will prevent SemrushBot from crawling your entire site. Our crawler can handle robots.txt files with a 3xx status code." Semrush notes that "it may take up to one hour or 100 requests for SemrushBot to discover changes made to your robots.txt", and Knowledge Base article 1056 shows a Site Audit example "User-agent: SemrushBot / Crawl-delay: 5" and says "the maximum crawl delay we can apply is 30" for that tool. The support contact is bot@semrush.com; JavaScript execution is not documented for this token.
Controlling SemrushBot with robots.txt
Use the token SemrushBot in robots.txt. Semrush documents that SemrushBot honors robots.txt directives.
User-agent: SemrushBot
Disallow: /User-agent: SemrushBot
Allow: /Verifying a request is really SemrushBot
Anyone can put SemrushBot in a User-Agent header. The operator does not document a verification method. Semrush states: "Do not try to block SemrushBot via IP as we do not use any consecutive IP blocks." No reverse-DNS suffix and no IP range file are published.
Common questions
Should I block SemrushBot?
Blocking the SemrushBot token stops Semrush from crawling your site for its webgraph of links, so your link data disappears from Backlink Analytics and the Link Building and Topic Research reports Semrush names. SemrushBot feeds Semrush's own tools rather than a search engine index, per the operator's description. Use robots.txt; Semrush asks you not to block by IP.
How do I verify SemrushBot?
Semrush documents no verification method. It publishes no reverse-DNS suffix and no IP range file, and states "Do not try to block SemrushBot via IP as we do not use any consecutive IP blocks." The only documented control is a robots.txt rule addressed to the SemrushBot token; questions go to bot@semrush.com.
Does SemrushBot follow Crawl-delay?
Yes, as a non-standard extension. SemrushBot for Backlink Analytics accepts intervals up to 10 seconds between requests and cuts higher values to that limit. Without a Crawl-delay it adjusts frequency to server load. A separate Site Audit article shows "Crawl-delay: 5" and says the maximum for that tool is 30.
Sources
Every fact on this page was checked against Semrush's own documentation, listed below, and re-checked by a second reviewer before publication. Reviewed September 2026.
Related crawlers
This registry documents how operators describe their own bots so site owners can identify and control them. It does not publish third-party IP lists or guess at undocumented behaviour. To see how your own site responds to automated visitors, the bot detection scanner reads a URL's live response and names the protection it finds.