SemrushBot-FT user agent, IP ranges and robots.txt

Semrush operates SemrushBot-FT to crawl sites for its Plagiarism Checker and similar tools, checking whether content being examined appears on a given website. Site owners control it through the SemrushBot-FT robots.txt token, which Semrush documents as honored. Semrush documents no verification method for it, stating that it uses no consecutive IP blocks, and documents no reverse-DNS check.

Semrush documents SemrushBot-FT only as a robots.txt User-agent token on its bot page, with a Disallow: / example for blocking it. Semrush publishes no full user-agent string for this crawler, so in logs you should expect the SemrushBot-FT token but cannot match against a documented complete string. Semrush says the crawler works for the Plagiarism Checker and similar tools. When someone checks a piece of text, the bot visits your site to verify whether that text appears there.

Allowing SemrushBot-FT lets Semrush's Plagiarism Checker confirm whether checked content appears on your pages. Blocking it with a Disallow rule under the SemrushBot-FT user agent stops those verification visits, so Semrush's tool can no longer confirm matches against your site. Semrush's page gives Disallow: / as the example, which refuses the whole site. Semrush documents the token as honored, and the effect it describes is limited to the Plagiarism Checker and similar tools.

There is no documented way to verify that a request claiming to be SemrushBot-FT is genuine. Semrush states that it does not use any consecutive IP blocks, documents no IP range list and documents no reverse-DNS hostname. Semrush also does not document whether the crawler executes JavaScript. Site owners can rely on the robots.txt token as a control, but not as proof of origin.

Operator note. Semrush documents this token only as a robots.txt User-agent that can be blocked with "Disallow: /" on its bot page. JavaScript execution and IP addresses are not documented.