Bingbot

What Bingbot does

Operator note. Microsoft documents three user agent strings: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/W.X.Y.Z Safari/537.36 for desktop rendering, Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) as the plain form, and Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) for mobile, noting "We regularly update our web page rendering engine to the most recent stable version of Microsoft Edge. Thus, 'W.X.Y.Z' will be substituted with the latest Microsoft Edge version we are using." On robots.txt, the documentation states "Bingbot honors robots.txt directives, including the crawl-delay: setting, and, in the absence of a crawl-delay, respects the input from webmasters in the Crawl Control feature", that "If we find a crawl-delay directive in your robots.txt file, then it will always take precedence" over Crawl Control, that crawl-delay accepts 1-20 seconds with one URL fetched per window (a crawl-delay of 5 allows 17280 URLs per day), and that "On finding a specific set of instructions for itself, Bingbot will ignore the directives listed in the generic section" of robots.txt. For robots meta tags and X-Robots-Tag, the generic name="robots" can be replaced with name="bingbot" to apply a directive only to Bing; supported values are noindex, noarchive, nocache, nosnippet, data-nosnippet, max-snippet, max-image-preview and max-video-preview, where noindex and noarchive also mean "Do not use the content for training Microsoft's generative AI foundation models" and nocache means "Use only URLs, Titles and Snippets for training" and "Display only URL/Snippet/Title in Chat or Copilot". Bingbot has not crawled sites that only support RC4 cipher suites since Autumn 2015 and the documentation publishes a supported cipher list for TLS 1.0, 1.1 and 1.2; the Verifying Bingbot article also refers to requests using a "Bingbot" or "MSNBOT" user agent string, though MSNBOT is not listed as a current crawler.

Controlling Bingbot with robots.txt

Use the token Bingbot in robots.txt. Microsoft Bing documents that Bingbot honors robots.txt directives.

User-agent: Bingbot
Disallow: /

User-agent: Bingbot
Allow: /

Verifying a request is really Bingbot

Anyone can put bingbot/2.0 in a User-Agent header. A reverse DNS lookup of the IP must resolve to a hostname ending in search.msn.com, and a forward DNS lookup of that hostname must return the same IP (the documentation's example hostname is msnbot-157-55-33-18.search.msn.com). Alternatively, match the IP against the published list at https://www.bing.com/toolbox/bingbot.json ("Refresh this list daily as we can change this list any time"), or use the Verify Bingbot tool, available publicly at http://www.bing.com/toolbox/verify-bingbot or inside Bing Webmaster Tools under Tools & Enhancements. The documentation carries the heading "Don't Use Hardcoded IP Addresses or Address Ranges".

host <request-ip>
# expect a hostname ending in search.msn.com
host <that-hostname>
# expect it to resolve back to <request-ip>

Common questions

Related crawlers