SemrushBot-ESI
Semrush operates SemrushBot-ESI, a crawler for its Enterprise Site Intelligence product, which analyzes on-page SEO, technical and usability issues. Site owners control it with the robots.txt token SemrushBot-ESI, which Semrush says it honors. Semrush documents no verification method, states it uses no consecutive IP blocks, and publishes no IP list or reverse-DNS suffix.
What SemrushBot-ESI does
SemrushBot-ESI is the user-agent token Semrush assigns to crawling done for its Enterprise Site Intelligence product. Semrush publishes no full user-agent string for it, so the token SemrushBot-ESI is the only string the operator gives a site owner to match in the agent field. The operator describes the product as one that analyzes on-page SEO, technical and usability issues, which means the visits exist to audit the site for the Semrush Enterprise Site Intelligence service.
Semrush's bot page documents this agent in one way only, as a robots.txt User-agent line paired with Disallow: /, given for site owners who want to stop the Enterprise Site Intelligence crawling. Semrush says the bot honors robots.txt, so that rule is the documented control. Blocking it stops Semrush from auditing the site's on-page SEO, technical and usability issues through this product. The operator draws no connection between this bot and search engines, so it describes no effect on search results from blocking it.
There is no documented way to verify a request claiming to be SemrushBot-ESI. Semrush states that it does not use any consecutive IP blocks, publishes no address list, and names no reverse-DNS suffix. JavaScript execution is also undocumented. A site owner is left with the token in the user-agent string and the robots.txt rule as the only tools the operator describes.
Operator note. Semrush documents this token only as a robots.txt User-agent that can be blocked with "Disallow: /" on its bot page. JavaScript execution and IP addresses are not documented.
Controlling SemrushBot-ESI with robots.txt
Use the token SemrushBot-ESI in robots.txt. Semrush documents that SemrushBot-ESI honors robots.txt directives.
User-agent: SemrushBot-ESI
Disallow: /User-agent: SemrushBot-ESI
Allow: /Verifying a request is really SemrushBot-ESI
Anyone can put SemrushBot-ESI in a User-Agent header. The operator does not document a verification method. Semrush states that "we do not use any consecutive IP blocks".
Common questions
Should I block SemrushBot-ESI?
If you do not want Semrush Enterprise Site Intelligence auditing your site's on-page SEO, technical and usability issues, yes. Semrush documents the block itself, a robots.txt User-agent line for SemrushBot-ESI with Disallow: /. Semrush describes the bot as part of an SEO product and draws no link to search engines, so the operator describes no search-visibility cost.
How do I verify SemrushBot-ESI?
You cannot, using anything Semrush documents. The operator says it does not use consecutive IP blocks, publishes no IP list, and names no reverse-DNS hostname. The only identifier is the SemrushBot-ESI token in the user-agent string, which on its own does not prove where a request came from.
Does SemrushBot-ESI run JavaScript?
The operator's documentation does not say. Semrush's bot page lists SemrushBot-ESI only as a robots.txt token with a Disallow example and gives no information about JavaScript execution, crawl rate, or the full user-agent string it sends.
Sources
Every fact on this page was checked against Semrush's own documentation, listed below, and re-checked by a second reviewer before publication. Reviewed September 2026.
Related crawlers
This registry documents how operators describe their own bots so site owners can identify and control them. It does not publish third-party IP lists or guess at undocumented behaviour. To see how your own site responds to automated visitors, the bot detection scanner reads a URL's live response and names the protection it finds.