SiteAuditBot
SiteAuditBot is the crawler behind Semrush's Site Audit tool, which analyzes on-page SEO, technical and usability issues; by default it crawls as SiteAuditBot (Mobile). It honors a Disallow under the SiteAuditBot token, but a verified site owner can turn on an option that bypasses robots.txt. Semrush publishes the subnet 85.208.98.128/25 in prose for verification and no reverse DNS suffix.
What SiteAuditBot does
Requests from Site Audit carry the SiteAuditBot user agent token. Semrush lets the person running the audit choose between SiteAuditBot (Desktop), SiteAuditBot (Mobile) and OpenAI-Search as the agent, and the mobile option is the default, which Semrush says audits the site the way Google's mobile crawler would. Semrush notes that the full string updates in its interface when the agent changes and can be copied into a cURL request, but it does not print that string in its public documentation. Connections use standard ports 80 and 443 and come from the subnet 85.208.98.128/25, which Semrush says Site Audit alone uses.
The visits exist because a Semrush user set up an audit of the site, so the crawl feeds that user's SEO report. Blocking it in robots.txt with User-agent: SiteAuditBot and Disallow: / stops ordinary audits, and Semrush documents the reverse form with an empty Disallow to allow them. The exception is a Site Audit option called Bypass disallow rules in robots.txt and meta robots tag, which Semrush only enables after the user proves ownership by uploading a Semrush-provided .txt file to the site root. A robots.txt block therefore holds against third parties but not against the site's own verified owner.
Semrush documents crawl rate and rendering. At the minimum setting the bot crawls at its normal rate, waiting around one second before the next page; Semrush also lists a rate of one URL per two seconds, and its Respect robots.txt setting makes the bot follow a crawl delay from robots.txt, with 30 as the maximum delay Semrush can apply. JavaScript rendering is optional and available with Guru or Business subscriptions; with it off, Site Audit reads only the HTML. Semrush says its crawlers cannot process a landing page, or a JavaScript and CSS total, larger than 2 MB. Audits can also run with the user's login credentials or with a Web Bot Auth signature.
Operator note. The Semrush bot page documents blocking with "User-agent: SiteAuditBot / Disallow: /" and Knowledge Base article 681 documents allowing with "User-agent: SiteAuditBot / Disallow:", but articles 539 and 681 also document a user option "Bypass disallow rules in robots.txt and meta robots tag": "When this option is turned on, the crawler will bypass robots.txt disallow rules ... Keep in mind that to use this, site ownership will have to be verified" (verified by uploading a .txt file Semrush provides to the site root). Crawl delay is described in article 539 ("Minimum: SiteAuditBot crawls at its normal rate, waiting around 1 second before crawling the next page"; "1 URL per 2 seconds"; "Respect robots.txt: SiteAuditBot follows the crawl delay specified in your robots.txt file") and article 1056 adds "the maximum crawl delay we can apply is 30". On JavaScript, article 681 says "Website content built on JavaScript - while Site Audit can render JS code, it can still be the reason for some of the issues" and "If your landing page size or the total size of JavaScript/CSS files exceeds 2Mb, our crawlers will be unable to process it", while article 539 describes optional rendering, "JS rendering: Turn this on to have SiteAuditBot execute your JavaScript files ... With JS rendering off, Site Audit reads only your HTML", which is "available with Guru or Business SEO Toolkit subscriptions". Other options are "Crawl with my credentials" (username and password for protected areas) and "Crawl with Web Bot Auth signature" ("A Web Bot Auth signature lets SiteAuditBot identify itself and prove it's authorized to access your site", configured per campaign with Signature Agent, Signature Input and Signature; no Semrush key directory URL is published); article 539 says "When you change the user agent, the string below the field updates to match. You can copy this string into a cURL request", although the full user-agent string is not printed in the documentation, and article 375 also says to whitelist "85.208.98.128/25 (a subnet used by Site Audit only) User-agent name: SiteAuditBot".
Controlling SiteAuditBot with robots.txt
Use the token SiteAuditBot in robots.txt. Semrush documents limits on how this bot applies robots.txt; see the operator note.
User-agent: SiteAuditBot
Disallow: /User-agent: SiteAuditBot
Allow: /Verifying a request is really SiteAuditBot
Anyone can put SiteAuditBot in a User-Agent header. Semrush publishes an IP range in Knowledge Base prose rather than a machine-readable file: "85.208.98.128/25 (a subnet used by Site Audit only)". "The bot is using standard 80 HTTP and 443 HTTPS ports to connect." No reverse-DNS suffix is documented.
Common questions
Should I block SiteAuditBot?
If you do not want third parties auditing your site through Semrush, add User-agent: SiteAuditBot with Disallow: / to robots.txt. Semrush honors that for ordinary audits. It does not stop a verified owner who has proven control of the domain and switched on the bypass option, so the block applies to outsiders while a verified owner can still audit.
Does SiteAuditBot run JavaScript?
Only when the person running the audit turns on JS rendering, which Semrush makes available with Guru or Business SEO Toolkit subscriptions. With rendering off, Site Audit reads only the HTML. Semrush also warns that a page, or its combined JavaScript and CSS, larger than 2 MB cannot be processed by its crawlers.
How do I verify SiteAuditBot?
Semrush publishes one subnet, 85.208.98.128/25, and says it is used by Site Audit only; the bot connects over standard ports 80 and 443. There is no machine-readable IP file and no reverse DNS suffix documented, so check the source IP against that subnet directly. Semrush also mentions an optional Web Bot Auth signature that lets the bot prove it is authorized, but publishes no directory URL for checking those signatures.
Sources
Every fact on this page was checked against Semrush's own documentation, listed below, and re-checked by a second reviewer before publication. Reviewed September 2026.
- https://www.semrush.com/bot/
- https://www.semrush.com/kb/681-site-audit-troubleshooting
- https://www.semrush.com/kb/539-configuring-site-audit
- https://www.semrush.com/kb/375-why-did-i-get-a-page-is-not-accessible-note-for-some-of-my-pages
- https://www.semrush.com/kb/1056-optimize-site-audit-crawl-speed
Related crawlers
This registry documents how operators describe their own bots so site owners can identify and control them. It does not publish third-party IP lists or guess at undocumented behaviour. To see how your own site responds to automated visitors, the bot detection scanner reads a URL's live response and names the protection it finds.