Amazonbot

Amazon publishes an IP address list for Amazonbot at https://developer.amazon.com/amazonbot/ip-addresses/, linked from the documentation as 'Published IP Addresses' and embedded as JSON within an HTML page rather than served as a raw JSON endpoint. Entries are single IPv4 addresses without a CIDR suffix; no IPv6 entries are listed. No reverse DNS suffix is documented.


ClaudeBot

Anthropic publishes a single shared IP range list for all of its bots at https://claude.com/crawling/bots.json and states: "If a crawler has a source IP address on this list, it indicates that the crawler is coming from Anthropic." The list is not broken down per bot, and no reverse-DNS suffix is documented.


Applebot-Extended

There is no separate traffic to verify: Apple states 'Applebot-Extended does not crawl webpages', so it has no user agent string or IP range of its own. It is a robots.txt policy token only, and the crawling itself is done by Applebot.


Google-Extended

Google-Extended has no crawler of its own to verify: 'Google-Extended doesn't have a separate HTTP request user agent string. Crawling is done with existing Google user agent strings; the robots.txt user-agent token is used in a control capacity.' Verify the underlying Google crawler (for example Googlebot) instead.


FacebookBot

Meta's retired FacebookBot documentation states: "Webmasters can easily identify traffic originating from the FacebookBot, either via its User-Agent string, or its originating IP addresses" and "You may also obtain the current list of IP addresses used by FacebookBot by running the following command: whois -h whois.radb.net -- '-i origin AS32934' | grep ^route". No reverse-DNS method is documented.


Meta-ExternalAgent

Meta does not document a verification method for this crawler. The web crawlers page gives only a page-wide statement to allow-list "the user agent strings or the IP addresses (more secure)", with no IP file, ASN, or reverse-DNS suffix given.


MistralAI-Training

The operator does not document a verification method.


GPTBot

OpenAI publishes IP ranges for GPTBot at https://openai.com/gptbot.json. No reverse-DNS method is documented.


Amzn-SearchBot

Amazon publishes an IP address list for Amzn-SearchBot at https://developer.amazon.com/amazonbot/searchbot-ip-addresses/, embedded as JSON within an HTML page. Entries are IPv4 addresses with a /32 CIDR suffix; no IPv6 entries are listed. No reverse DNS suffix is documented.


Claude-SearchBot

Anthropic publishes a single shared IP range list for all of its bots at https://claude.com/crawling/bots.json. The list is not broken down per bot, and no reverse-DNS suffix is documented.


DuckAssistBot

Verification is by published IP list only. The help page lists the originating IP addresses inline and links to the machine-readable list at https://duckduckgo.com/duckassistbot.json. The published list contains IPv4 host addresses only, and no reverse-DNS suffix is documented.


Google-CloudVertexBot

googlebot.com (crawl-***-***-***-***.googlebot.com or geo-crawl-***-***-***-***.geo.googlebot.com)

As a listed common crawler, it is verified by reverse DNS to googlebot.com, google.com, or googleusercontent.com with matching forward DNS, or by matching the source IP against the published common-crawlers.json ranges.


Meta-WebIndexer

Meta does not document a verification method for this crawler. The web crawlers page gives only a page-wide statement: "Add to your allow list either the user agent strings or the IP addresses (more secure) used by the crawler." No IP file, ASN, or reverse-DNS suffix is given on the crawler page; Meta's AS32934 whois command and geofeed CSV are published only in the context of its webhooks documentation.


MistralAI-Index

Mistral publishes the IP ranges for this crawler at https://mistral.ai/mistralai-index-ips.json. No reverse-DNS method is documented.


OAI-SearchBot

OpenAI publishes IP ranges for OAI-SearchBot at https://openai.com/searchbot.json. The documentation says: "we recommend allowing OAI-SearchBot in your site's robots.txt file and allowing requests from our published IP ranges below." No reverse-DNS method is documented.


PerplexityBot

The operator publishes IP ranges only. The docs say to combine User-Agent string matching with IP address verification against the published JSON range file. No reverse DNS method is documented.


YandexAdditionalBot

Reverse DNS lookup on the requesting IP must return a host ending in yandex.ru, yandex.net or yandex.com, and a forward DNS lookup of that host must return the same IP. Yandex Webmaster also provides an IP address check tool.


Amzn-User

Amazon publishes an IP address list for Amzn-User at https://developer.amazon.com/amazonbot/live-ip-addresses/, embedded as JSON within an HTML page. Entries are single IPv4 addresses without a CIDR suffix; no IPv6 entries are listed. No reverse DNS suffix is documented.


Chrome Web Store

gae.googleusercontent.com (***-***-***-***.gae.googleusercontent.com) or google.com (google-proxy-***-***-***-***.google.com)

Reverse DNS with matching forward DNS, or match the source IP against user-triggered-fetchers.json or user-triggered-fetchers-google.json.


Feedfetcher

Reverse DNS with matching forward DNS, or match the source IP against user-triggered-fetchers.json (user-owned, resolves to gae.googleusercontent.com) or user-triggered-fetchers-google.json (Google-controlled, resolves to google.com).


Google-Agent

undocumented for agents specifically; the user-triggered fetchers page gives ***-***-***-***.gae.googleusercontent.com or google-proxy-***-***-***-***.google.com for the category

Match the source IP against user-triggered-agents.json (the docs say 'It uses IP ranges from user-triggered-agents.json'), together with the general reverse and forward DNS check. A subset of Google-Agent requests also carry an experimental Web Bot Auth signature: Signature-Agent: g="https://agent.bot.goog", signed per RFC 9421 HTTP Message Signatures, with public keys at https://agent.bot.goog/.well-known/http-message-signatures-directory. Google says not all requests are signed and to fall back to IP-based verification.


MistralAI-User

Mistral publishes the IP ranges for this crawler at https://mistral.ai/mistralai-user-ips.json. No reverse-DNS method is documented.


ChatGPT-User

OpenAI publishes IP ranges for ChatGPT-User at https://openai.com/chatgpt-user.json. No reverse-DNS method is documented.


Applebot

Apple documents two methods. Reverse DNS: the IP should resolve to a host within *.applebot.apple.com, and a forward lookup of that hostname should return the same IP (Apple's example: 17.58.101.179 resolves to 17-58-101-179.applebot.apple.com, which resolves back to 17.58.101.179). IP matching: compare the address against the CIDR prefixes published at https://search.developer.apple.com/applebot.json.


Bytespider

ByteDance's official documentation says webmasters can verify Bytespider by reverse DNS lookup (host, nslookup -qt=ptr, or dig -x): genuine Bytespider hostnames follow the *.bytedance.com pattern, and anything else is an impostor (冒充). The documentation gives the worked example host 111.225.148.250 -> bytespider-111-225-148-148.crawl.bytedance.com. The same page lists ten IPv4 /24 ranges inline (110.249.201.0/24, 110.249.202.0/24, 111.225.148.0/24, 111.225.149.0/24, 220.243.135.0/24, 220.243.136.0/24, 220.243.188.0/24, 220.243.189.0/24, 60.8.123.0/24, 60.8.151.0/24); no machine-readable JSON or TXT IP range file is published.


DuckDuckBot

Verification is by published IP list only. The help page lists the originating IP addresses inline and links to the machine-readable list at https://duckduckgo.com/duckduckbot.json. The published list contains IPv4 host addresses only, and no reverse-DNS suffix is documented.


Google StoreBot

Reverse DNS to googlebot.com, google.com, or googleusercontent.com with matching forward DNS, or match the source IP against the published common-crawlers.json ranges.


Googlebot

A reverse DNS lookup of the source IP must resolve to googlebot.com, google.com, or googleusercontent.com, and a forward DNS lookup of that name must return the same IP. Alternatively, match the IP against the published common-crawlers.json CIDR ranges.


BingVideoPreview

Microsoft does not document a verification method specific to BingVideoPreview. The generic guidance for all Bing crawlers applies: use the Verify Bingbot tool, check that reverse DNS resolves to a hostname under search.msn.com, or match the IP against the published list at https://www.bing.com/toolbox/bingbot.json.


Bingbot

A reverse DNS lookup of the IP must resolve to a hostname ending in search.msn.com, and a forward DNS lookup of that hostname must return the same IP (the documentation's example hostname is msnbot-157-55-33-18.search.msn.com). Alternatively, match the IP against the published list at https://www.bing.com/toolbox/bingbot.json ("Refresh this list daily as we can change this list any time"), or use the Verify Bingbot tool, available publicly at http://www.bing.com/toolbox/verify-bingbot or inside Bing Webmaster Tools under Tools & Enhancements. The documentation carries the heading "Don't Use Hardcoded IP Addresses or Address Ranges".


YandexBot

Reverse DNS lookup on the requesting IP must return a host ending in yandex.ru, yandex.net or yandex.com, and a forward DNS lookup of that host must return the same IP. Yandex Webmaster also offers an 'IP address check' tool (Tools -> IP address check). Yandex bots use AS13238, AS208722 and AS212066 with frequently changing IPs, and the IP list is not disclosed.


FacebookExternalHit

Meta's current crawler page gives no method beyond "Add to your allow list either the user agent strings or the IP addresses (more secure) used by the crawler." The retired Facebook Crawler page documented: "To get a current list of IP addresses the crawler uses, run the following command. whois -h whois.radb.net -- '-i origin AS32934' | grep ^route" and "These IP addresses change often." Meta currently publishes a variant of that command on its WhatsApp webhooks page, described as returning "the IP addresses of Meta's webhook servers" rather than crawlers: "whois -h whois.radb.net -- '-i origin AS32934' | grep '^route' | awk '{print $2}' | sort", and links a geofeed CSV at https://facebook.com/peering/geofeed ("download a CSV that lists Meta's IP addresses"); no reverse-DNS method is documented.


MicrosoftPreview

Microsoft does not document a verification method specific to MicrosoftPreview. The generic guidance for all Bing crawlers applies: use the Verify Bingbot tool, check that reverse DNS resolves to a hostname under search.msn.com, or match the IP against the published list at https://www.bing.com/toolbox/bingbot.json.


AhrefsBot

Ahrefs publishes its crawler IP ranges at https://api.ahrefs.com/v3/public/crawler-ip-ranges and individual crawler IP addresses at https://api.ahrefs.com/v3/public/crawler-ips. Reverse DNS can also be used: 'The reverse DNS suffix of the IPs hostname is always ahrefs.com or ahrefs.net.' AhrefsBot is also listed as a Cloudflare verified 'good' bot.


AhrefsSiteAudit

AhrefsSiteAudit uses the same published IP lists as AhrefsBot: https://api.ahrefs.com/v3/public/crawler-ip-ranges and https://api.ahrefs.com/v3/public/crawler-ips. The reverse DNS suffix of its hostnames is 'always ahrefs.com or ahrefs.net'. It is also listed as a Cloudflare verified 'good' bot.


RyteBot

The operator does not document a verification method.


SemrushBot

The operator does not document a verification method. Semrush states: "Do not try to block SemrushBot via IP as we do not use any consecutive IP blocks." No reverse-DNS suffix and no IP range file are published.


SemrushBot-BA

The operator does not document a verification method. Semrush states that "we do not use any consecutive IP blocks".


SemrushBot-SI

Semrush publishes IP addresses in Knowledge Base prose rather than a machine-readable file: article 375 says to "whitelist the following IP addresses and User-agent", listing "85.208.98.53", "85.208.98.0/24" and "User-agent: SemrushBot-SI", on "Port 80: HTTP" and "Port 443: HTTPS". No reverse-DNS suffix is documented.


SiteAuditBot

Semrush publishes an IP range in Knowledge Base prose rather than a machine-readable file: "85.208.98.128/25 (a subnet used by Site Audit only)". "The bot is using standard 80 HTTP and 443 HTTPS ports to connect." No reverse-DNS suffix is documented.


YandexPagechecker

Reverse DNS lookup on the requesting IP must return a host ending in yandex.ru, yandex.net or yandex.com, and a forward DNS lookup of that host must return the same IP. Yandex Webmaster also provides an IP address check tool.


CCBot

Common Crawl publishes its IP ranges at https://index.commoncrawl.org/ccbot.json and lists the same CIDRs inline in its FAQ, where the IPv6 range is labelled "IPv6 (no reverse DNS)"; the file's notes read: "IP ranges used by CCBot. For verification of IPv4 addresses, FCrDNS is also recommended." The operator states: "CCBot is now run on dedicated IP address ranges with reverse DNS (except over IPv6 where reverse DNS is not yet supported.)" and gives the example that 18.97.14.84 resolves to 18-97-14-84.crawl.commoncrawl.org, which resolves back to 18.97.14.84. The documentation warns: "we are aware of crawlers falsely identifying themselves as CCBot. We recommend verifying UserAgent strings to ensure authenticity."


iTMS

Apple says iTMS traffic 'may come from applebot.apple.com hosts', so the Applebot reverse-DNS check applies to that traffic. No dedicated IP range file is documented for iTMS.


APIs-Google

Reverse DNS to rate-limited-proxy-***-***-***-***.google.com with matching forward DNS, or match the source IP against special-crawlers.json.


GoogleOther

Reverse DNS to googlebot.com, google.com, or googleusercontent.com with matching forward DNS, or match the source IP against the published common-crawlers.json ranges.


Meta-ExternalAds

Meta does not document a verification method for this crawler. The web crawlers page gives only a page-wide statement to allow-list "the user agent strings or the IP addresses (more secure)", with no IP file, ASN, or reverse-DNS suffix given.


AdIdxBot

Microsoft does not document a verification method specific to AdIdxBot. The crawlers article states for all Bing crawlers: "To ensure the IP is authentic, use the Verify Bingbot tool or learn about all verification methods we offer", those methods being reverse DNS to search.msn.com and the published list at https://www.bing.com/toolbox/bingbot.json. Whether AdIdxBot IPs are included in bingbot.json or resolve under search.msn.com is not stated.


OAI-AdsBot

OpenAI publishes IP ranges for OAI-AdsBot at https://openai.com/adsbot.json. No reverse-DNS method is documented.