SEO tool crawler · Semrush
SemrushBot-ESI user agent, IP ranges and robots.txt
SemrushBot-ESI is Semrush's SEO tool crawler. It identifies itself with the SemrushBot-ESI user-agent token, honors robots.txt according to its operator, and has no official IP range file.
- Operator
- Semrush
- Reviewed
- September 2026
- Sources
- 1
- Operator
- Semrush
- User-agent token
- SemrushBot-ESI
- Purpose
- SEO tool crawler
- robots.txt token
- SemrushBot-ESI
- Honors robots.txt
- Yes, per the operator
- Published IP ranges
- No official range file
- Verification
- The operator does not document a verification method. Semrush states that "we do not use any consecutive IP blocks".
- Documentation
- www.semrush.com/bot/
Block or allow SemrushBot-ESI in robots.txt
User-agent: SemrushBot-ESI
Disallow: /User-agent: SemrushBot-ESI
Allow: /Verify a request is really SemrushBot-ESI
Anyone can put SemrushBot-ESI in a User-Agent header. The operator does not document a verification method. Semrush states that "we do not use any consecutive IP blocks".
What SemrushBot-ESI does
Semrush operates SemrushBot-ESI, a crawler for its Enterprise Site Intelligence product, which analyzes on-page SEO, technical and usability issues. Site owners control it with the robots.txt token SemrushBot-ESI, which Semrush says it honors. Semrush documents no verification method, states it uses no consecutive IP blocks, and publishes no IP list or reverse-DNS suffix.
SemrushBot-ESI is the user-agent token Semrush assigns to crawling done for its Enterprise Site Intelligence product. Semrush publishes no full user-agent string for it, so the token SemrushBot-ESI is the only string the operator gives a site owner to match in the agent field. The operator describes the product as one that analyzes on-page SEO, technical and usability issues, which means the visits exist to audit the site for the Semrush Enterprise Site Intelligence service.
Semrush's bot page documents this agent in one way only, as a robots.txt User-agent line paired with Disallow: /, given for site owners who want to stop the Enterprise Site Intelligence crawling. Semrush says the bot honors robots.txt, so that rule is the documented control. Blocking it stops Semrush from auditing the site's on-page SEO, technical and usability issues through this product. The operator draws no connection between this bot and search engines, so it describes no effect on search results from blocking it.
There is no documented way to verify a request claiming to be SemrushBot-ESI. Semrush states that it does not use any consecutive IP blocks, publishes no address list, and names no reverse-DNS suffix. JavaScript execution is also undocumented. A site owner is left with the token in the user-agent string and the robots.txt rule as the only tools the operator describes.
Operator note. Semrush documents this token only as a robots.txt User-agent that can be blocked with "Disallow: /" on its bot page. JavaScript execution and IP addresses are not documented.
Common questions
- Should I block SemrushBot-ESI?
- If you do not want Semrush Enterprise Site Intelligence auditing your site's on-page SEO, technical and usability issues, yes. Semrush documents the block itself, a robots.txt User-agent line for SemrushBot-ESI with Disallow: /. Semrush describes the bot as part of an SEO product and draws no link to search engines, so the operator describes no search-visibility cost.
- How do I verify SemrushBot-ESI?
- You cannot, using anything Semrush documents. The operator says it does not use consecutive IP blocks, publishes no IP list, and names no reverse-DNS hostname. The only identifier is the SemrushBot-ESI token in the user-agent string, which on its own does not prove where a request came from.
- Does SemrushBot-ESI run JavaScript?
- The operator's documentation does not say. Semrush's bot page lists SemrushBot-ESI only as a robots.txt token with a Disallow example and gives no information about JavaScript execution, crawl rate, or the full user-agent string it sends.
Sources
The operator documentation used for this entry. Recorded review date September 2026; consult the linked pages for current policy and compare them with this dated snapshot.
This registry documents how operators describe their own bots so site owners can identify and control them. It does not publish third-party IP lists or guess at undocumented behaviour. To see how your own site responds to automated visitors, the bot detection scanner reads a URL's live response and names the protection it finds.