{"slug":"amazonbot","name":"Amazonbot","operator":"Amazon","user_agent_token":"Amazonbot","purpose_category":"ai-training","robots_token":"Amazonbot","robots_token_is_policy_only":false,"honors_robots_txt":"yes","ip_ranges_url":"https://developer.amazon.com/amazonbot/ip-addresses/","ip_prefix_count":525,"ip_ranges_creation_time":"","reverse_dns_suffix":"","verification_method":"Amazon publishes an IP address list for Amazonbot at https://developer.amazon.com/amazonbot/ip-addresses/, linked from the documentation as 'Published IP Addresses' and embedded as JSON within an HTML page rather than served as a raw JSON endpoint. Entries are single IPv4 addresses without a CIDR suffix; no IPv6 entries are listed. No reverse DNS suffix is documented.","docs_url":"https://developer.amazon.com/amazonbot","page_url":"https://www.proxies.sx/crawlers/amazonbot"}
{"slug":"claudebot","name":"ClaudeBot","operator":"Anthropic","user_agent_token":"ClaudeBot","purpose_category":"ai-training","robots_token":"ClaudeBot","robots_token_is_policy_only":false,"honors_robots_txt":"yes","ip_ranges_url":"https://claude.com/crawling/bots.json","ip_prefix_count":26,"ip_ranges_creation_time":"2026-08-18T23:56:36Z","reverse_dns_suffix":"","verification_method":"Anthropic publishes a single shared IP range list for all of its bots at https://claude.com/crawling/bots.json and states: \"If a crawler has a source IP address on this list, it indicates that the crawler is coming from Anthropic.\" The list is not broken down per bot, and no reverse-DNS suffix is documented.","docs_url":"https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler","page_url":"https://www.proxies.sx/crawlers/claudebot"}
{"slug":"applebot-extended","name":"Applebot-Extended","operator":"Apple","user_agent_token":"Applebot-Extended","purpose_category":"ai-training","robots_token":"Applebot-Extended","robots_token_is_policy_only":true,"honors_robots_txt":"yes","ip_ranges_url":"","ip_prefix_count":"","ip_ranges_creation_time":"","reverse_dns_suffix":"","verification_method":"There is no separate traffic to verify: Apple states 'Applebot-Extended does not crawl webpages', so it has no user agent string or IP range of its own. It is a robots.txt policy token only, and the crawling itself is done by Applebot.","docs_url":"https://support.apple.com/en-us/119829","page_url":"https://www.proxies.sx/crawlers/applebot-extended"}
{"slug":"google-extended","name":"Google-Extended","operator":"Google","user_agent_token":"Google-Extended","purpose_category":"ai-training","robots_token":"Google-Extended","robots_token_is_policy_only":true,"honors_robots_txt":"yes","ip_ranges_url":"","ip_prefix_count":"","ip_ranges_creation_time":"","reverse_dns_suffix":"","verification_method":"Google-Extended has no crawler of its own to verify: 'Google-Extended doesn't have a separate HTTP request user agent string. Crawling is done with existing Google user agent strings; the robots.txt user-agent token is used in a control capacity.' Verify the underlying Google crawler (for example Googlebot) instead.","docs_url":"https://developers.google.com/crawling/docs/crawlers-fetchers/overview-google-crawlers","page_url":"https://www.proxies.sx/crawlers/google-extended"}
{"slug":"facebookbot","name":"FacebookBot","operator":"Meta","user_agent_token":"FacebookBot","purpose_category":"ai-training","robots_token":"FacebookBot","robots_token_is_policy_only":false,"honors_robots_txt":"yes","ip_ranges_url":"","ip_prefix_count":"","ip_ranges_creation_time":"","reverse_dns_suffix":"","verification_method":"Meta's retired FacebookBot documentation states: \"Webmasters can easily identify traffic originating from the FacebookBot, either via its User-Agent string, or its originating IP addresses\" and \"You may also obtain the current list of IP addresses used by FacebookBot by running the following command: whois -h whois.radb.net -- '-i origin AS32934' | grep ^route\". No reverse-DNS method is documented.","docs_url":"https://developers.facebook.com/docs/sharing/webmasters/web-crawlers","page_url":"https://www.proxies.sx/crawlers/facebookbot"}
{"slug":"meta-externalagent","name":"Meta-ExternalAgent","operator":"Meta","user_agent_token":"meta-externalagent","purpose_category":"ai-training","robots_token":"meta-externalagent","robots_token_is_policy_only":false,"honors_robots_txt":"yes","ip_ranges_url":"","ip_prefix_count":"","ip_ranges_creation_time":"","reverse_dns_suffix":"","verification_method":"Meta does not document a verification method for this crawler. The web crawlers page gives only a page-wide statement to allow-list \"the user agent strings or the IP addresses (more secure)\", with no IP file, ASN, or reverse-DNS suffix given.","docs_url":"https://developers.facebook.com/docs/sharing/webmasters/web-crawlers","page_url":"https://www.proxies.sx/crawlers/meta-externalagent"}
{"slug":"mistralai-training","name":"MistralAI-Training","operator":"Mistral AI","user_agent_token":"MistralAI-Training","purpose_category":"ai-training","robots_token":"MistralAI-Training","robots_token_is_policy_only":false,"honors_robots_txt":"yes","ip_ranges_url":"","ip_prefix_count":"","ip_ranges_creation_time":"","reverse_dns_suffix":"","verification_method":"The operator does not document a verification method.","docs_url":"https://docs.mistral.ai/robots","page_url":"https://www.proxies.sx/crawlers/mistralai-training"}
{"slug":"gptbot","name":"GPTBot","operator":"OpenAI","user_agent_token":"GPTBot","purpose_category":"ai-training","robots_token":"GPTBot","robots_token_is_policy_only":false,"honors_robots_txt":"yes","ip_ranges_url":"https://openai.com/gptbot.json","ip_prefix_count":21,"ip_ranges_creation_time":"2025-10-30T11:00:00.000000","reverse_dns_suffix":"","verification_method":"OpenAI publishes IP ranges for GPTBot at https://openai.com/gptbot.json. No reverse-DNS method is documented.","docs_url":"https://developers.openai.com/api/docs/bots","page_url":"https://www.proxies.sx/crawlers/gptbot"}
{"slug":"amzn-searchbot","name":"Amzn-SearchBot","operator":"Amazon","user_agent_token":"Amzn-SearchBot","purpose_category":"ai-search-index","robots_token":"Amzn-SearchBot","robots_token_is_policy_only":false,"honors_robots_txt":"yes","ip_ranges_url":"https://developer.amazon.com/amazonbot/searchbot-ip-addresses/","ip_prefix_count":513,"ip_ranges_creation_time":"","reverse_dns_suffix":"","verification_method":"Amazon publishes an IP address list for Amzn-SearchBot at https://developer.amazon.com/amazonbot/searchbot-ip-addresses/, embedded as JSON within an HTML page. Entries are IPv4 addresses with a /32 CIDR suffix; no IPv6 entries are listed. No reverse DNS suffix is documented.","docs_url":"https://developer.amazon.com/amazonbot","page_url":"https://www.proxies.sx/crawlers/amzn-searchbot"}
{"slug":"claude-searchbot","name":"Claude-SearchBot","operator":"Anthropic","user_agent_token":"Claude-SearchBot","purpose_category":"ai-search-index","robots_token":"Claude-SearchBot","robots_token_is_policy_only":false,"honors_robots_txt":"yes","ip_ranges_url":"https://claude.com/crawling/bots.json","ip_prefix_count":26,"ip_ranges_creation_time":"2026-08-18T23:56:36Z","reverse_dns_suffix":"","verification_method":"Anthropic publishes a single shared IP range list for all of its bots at https://claude.com/crawling/bots.json. The list is not broken down per bot, and no reverse-DNS suffix is documented.","docs_url":"https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler","page_url":"https://www.proxies.sx/crawlers/claude-searchbot"}
{"slug":"duckassistbot","name":"DuckAssistBot","operator":"DuckDuckGo","user_agent_token":"DuckAssistBot","purpose_category":"ai-search-index","robots_token":"DuckAssistBot","robots_token_is_policy_only":false,"honors_robots_txt":"yes","ip_ranges_url":"https://duckduckgo.com/duckassistbot.json","ip_prefix_count":486,"ip_ranges_creation_time":"2026-09-01T12:44:58.000000","reverse_dns_suffix":"","verification_method":"Verification is by published IP list only. The help page lists the originating IP addresses inline and links to the machine-readable list at https://duckduckgo.com/duckassistbot.json. The published list contains IPv4 host addresses only, and no reverse-DNS suffix is documented.","docs_url":"https://duckduckgo.com/duckduckgo-help-pages/results/duckduckbot/","page_url":"https://www.proxies.sx/crawlers/duckassistbot"}
{"slug":"google-cloudvertexbot","name":"Google-CloudVertexBot","operator":"Google","user_agent_token":"Google-CloudVertexBot","purpose_category":"ai-search-index","robots_token":"Google-CloudVertexBot","robots_token_is_policy_only":false,"honors_robots_txt":"yes","ip_ranges_url":"https://developers.google.com/static/crawling/ipranges/common-crawlers.json","ip_prefix_count":317,"ip_ranges_creation_time":"2026-09-04T14:46:55.000000","reverse_dns_suffix":"googlebot.com (crawl-***-***-***-***.googlebot.com or geo-crawl-***-***-***-***.geo.googlebot.com)","verification_method":"As a listed common crawler, it is verified by reverse DNS to googlebot.com, google.com, or googleusercontent.com with matching forward DNS, or by matching the source IP against the published common-crawlers.json ranges.","docs_url":"https://developers.google.com/crawling/docs/crawlers-fetchers/overview-google-crawlers","page_url":"https://www.proxies.sx/crawlers/google-cloudvertexbot"}
{"slug":"meta-webindexer","name":"Meta-WebIndexer","operator":"Meta","user_agent_token":"meta-webindexer","purpose_category":"ai-search-index","robots_token":"meta-webindexer","robots_token_is_policy_only":false,"honors_robots_txt":"yes","ip_ranges_url":"","ip_prefix_count":"","ip_ranges_creation_time":"","reverse_dns_suffix":"","verification_method":"Meta does not document a verification method for this crawler. The web crawlers page gives only a page-wide statement: \"Add to your allow list either the user agent strings or the IP addresses (more secure) used by the crawler.\" No IP file, ASN, or reverse-DNS suffix is given on the crawler page; Meta's AS32934 whois command and geofeed CSV are published only in the context of its webhooks documentation.","docs_url":"https://developers.facebook.com/docs/sharing/webmasters/web-crawlers","page_url":"https://www.proxies.sx/crawlers/meta-webindexer"}
{"slug":"mistralai-index","name":"MistralAI-Index","operator":"Mistral AI","user_agent_token":"MistralAI-Index","purpose_category":"ai-search-index","robots_token":"MistralAI-Index","robots_token_is_policy_only":false,"honors_robots_txt":"undocumented","ip_ranges_url":"https://mistral.ai/mistralai-index-ips.json","ip_prefix_count":2,"ip_ranges_creation_time":"2026-04-19T12:00:00.000000","reverse_dns_suffix":"","verification_method":"Mistral publishes the IP ranges for this crawler at https://mistral.ai/mistralai-index-ips.json. No reverse-DNS method is documented.","docs_url":"https://docs.mistral.ai/robots","page_url":"https://www.proxies.sx/crawlers/mistralai-index"}
{"slug":"oai-searchbot","name":"OAI-SearchBot","operator":"OpenAI","user_agent_token":"OAI-SearchBot","purpose_category":"ai-search-index","robots_token":"OAI-SearchBot","robots_token_is_policy_only":false,"honors_robots_txt":"yes","ip_ranges_url":"https://openai.com/searchbot.json","ip_prefix_count":35,"ip_ranges_creation_time":"2026-01-02T11:00:00.000000","reverse_dns_suffix":"","verification_method":"OpenAI publishes IP ranges for OAI-SearchBot at https://openai.com/searchbot.json. The documentation says: \"we recommend allowing OAI-SearchBot in your site's robots.txt file and allowing requests from our published IP ranges below.\" No reverse-DNS method is documented.","docs_url":"https://developers.openai.com/api/docs/bots","page_url":"https://www.proxies.sx/crawlers/oai-searchbot"}
{"slug":"perplexitybot","name":"PerplexityBot","operator":"Perplexity","user_agent_token":"PerplexityBot","purpose_category":"ai-search-index","robots_token":"PerplexityBot","robots_token_is_policy_only":false,"honors_robots_txt":"yes","ip_ranges_url":"https://www.perplexity.com/perplexitybot.json","ip_prefix_count":8,"ip_ranges_creation_time":"2025-02-07T16:56:00.000000","reverse_dns_suffix":"","verification_method":"The operator publishes IP ranges only. The docs say to combine User-Agent string matching with IP address verification against the published JSON range file. No reverse DNS method is documented.","docs_url":"https://docs.perplexity.ai/guides/bots","page_url":"https://www.proxies.sx/crawlers/perplexitybot"}
{"slug":"yandexadditionalbot","name":"YandexAdditionalBot","operator":"Yandex","user_agent_token":"YandexAdditionalBot","purpose_category":"ai-search-index","robots_token":"YandexAdditionalBot","robots_token_is_policy_only":true,"honors_robots_txt":"partial","ip_ranges_url":"","ip_prefix_count":"","ip_ranges_creation_time":"","reverse_dns_suffix":"yandex.ru, yandex.net, yandex.com","verification_method":"Reverse DNS lookup on the requesting IP must return a host ending in yandex.ru, yandex.net or yandex.com, and a forward DNS lookup of that host must return the same IP. Yandex Webmaster also provides an IP address check tool.","docs_url":"https://yandex.com/support/webmaster/en/robot-workings/check-yandex-robots.html","page_url":"https://www.proxies.sx/crawlers/yandexadditionalbot"}
{"slug":"amzn-user","name":"Amzn-User","operator":"Amazon","user_agent_token":"Amzn-User","purpose_category":"user-triggered-fetch","robots_token":"Amzn-User","robots_token_is_policy_only":false,"honors_robots_txt":"partial","ip_ranges_url":"https://developer.amazon.com/amazonbot/live-ip-addresses/","ip_prefix_count":1024,"ip_ranges_creation_time":"","reverse_dns_suffix":"","verification_method":"Amazon publishes an IP address list for Amzn-User at https://developer.amazon.com/amazonbot/live-ip-addresses/, embedded as JSON within an HTML page. Entries are single IPv4 addresses without a CIDR suffix; no IPv6 entries are listed. No reverse DNS suffix is documented.","docs_url":"https://developer.amazon.com/amazonbot","page_url":"https://www.proxies.sx/crawlers/amzn-user"}
{"slug":"claude-user","name":"Claude-User","operator":"Anthropic","user_agent_token":"Claude-User","purpose_category":"user-triggered-fetch","robots_token":"Claude-User","robots_token_is_policy_only":false,"honors_robots_txt":"yes","ip_ranges_url":"https://claude.com/crawling/bots.json","ip_prefix_count":26,"ip_ranges_creation_time":"2026-08-18T23:56:36Z","reverse_dns_suffix":"","verification_method":"Anthropic publishes a single shared IP range list for all of its bots at https://claude.com/crawling/bots.json. The list is not broken down per bot, and no reverse-DNS suffix is documented.","docs_url":"https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler","page_url":"https://www.proxies.sx/crawlers/claude-user"}
{"slug":"chrome-web-store","name":"Chrome Web Store","operator":"Google","user_agent_token":"Google-CWS","purpose_category":"user-triggered-fetch","robots_token":"undocumented","robots_token_is_policy_only":false,"honors_robots_txt":"no","ip_ranges_url":"https://developers.google.com/static/crawling/ipranges/user-triggered-fetchers-google.json","ip_prefix_count":496,"ip_ranges_creation_time":"2026-09-04T14:46:58.000000","reverse_dns_suffix":"gae.googleusercontent.com (***-***-***-***.gae.googleusercontent.com) or google.com (google-proxy-***-***-***-***.google.com)","verification_method":"Reverse DNS with matching forward DNS, or match the source IP against user-triggered-fetchers.json or user-triggered-fetchers-google.json.","docs_url":"https://developers.google.com/crawling/docs/crawlers-fetchers/overview-google-crawlers","page_url":"https://www.proxies.sx/crawlers/chrome-web-store"}
{"slug":"feedfetcher","name":"Feedfetcher","operator":"Google","user_agent_token":"FeedFetcher-Google","purpose_category":"user-triggered-fetch","robots_token":"undocumented","robots_token_is_policy_only":false,"honors_robots_txt":"no","ip_ranges_url":"https://developers.google.com/static/crawling/ipranges/user-triggered-fetchers-google.json","ip_prefix_count":496,"ip_ranges_creation_time":"2026-09-04T14:46:58.000000","reverse_dns_suffix":"gae.googleusercontent.com (***-***-***-***.gae.googleusercontent.com) or google.com (google-proxy-***-***-***-***.google.com)","verification_method":"Reverse DNS with matching forward DNS, or match the source IP against user-triggered-fetchers.json (user-owned, resolves to gae.googleusercontent.com) or user-triggered-fetchers-google.json (Google-controlled, resolves to google.com).","docs_url":"https://developers.google.com/crawling/docs/crawlers-fetchers/overview-google-crawlers","page_url":"https://www.proxies.sx/crawlers/feedfetcher"}
{"slug":"gemini-notebook","name":"Gemini Notebook","operator":"Google","user_agent_token":"Google-GeminiNotebook","purpose_category":"user-triggered-fetch","robots_token":"undocumented","robots_token_is_policy_only":false,"honors_robots_txt":"no","ip_ranges_url":"https://developers.google.com/static/crawling/ipranges/user-triggered-fetchers-google.json","ip_prefix_count":496,"ip_ranges_creation_time":"2026-09-04T14:46:58.000000","reverse_dns_suffix":"gae.googleusercontent.com (***-***-***-***.gae.googleusercontent.com) or google.com (google-proxy-***-***-***-***.google.com)","verification_method":"Reverse DNS with matching forward DNS, or match the source IP against user-triggered-fetchers.json or user-triggered-fetchers-google.json.","docs_url":"https://developers.google.com/crawling/docs/crawlers-fetchers/overview-google-crawlers","page_url":"https://www.proxies.sx/crawlers/gemini-notebook"}
{"slug":"google-pinpoint","name":"Google Pinpoint","operator":"Google","user_agent_token":"Google-Pinpoint","purpose_category":"user-triggered-fetch","robots_token":"undocumented","robots_token_is_policy_only":false,"honors_robots_txt":"no","ip_ranges_url":"https://developers.google.com/static/crawling/ipranges/user-triggered-fetchers-google.json","ip_prefix_count":496,"ip_ranges_creation_time":"2026-09-04T14:46:58.000000","reverse_dns_suffix":"gae.googleusercontent.com (***-***-***-***.gae.googleusercontent.com) or google.com (google-proxy-***-***-***-***.google.com)","verification_method":"Reverse DNS with matching forward DNS, or match the source IP against user-triggered-fetchers.json or user-triggered-fetchers-google.json.","docs_url":"https://developers.google.com/crawling/docs/crawlers-fetchers/overview-google-crawlers","page_url":"https://www.proxies.sx/crawlers/google-pinpoint"}
{"slug":"google-publisher-center","name":"Google Publisher Center","operator":"Google","user_agent_token":"GoogleProducer","purpose_category":"user-triggered-fetch","robots_token":"undocumented","robots_token_is_policy_only":false,"honors_robots_txt":"no","ip_ranges_url":"https://developers.google.com/static/crawling/ipranges/user-triggered-fetchers-google.json","ip_prefix_count":496,"ip_ranges_creation_time":"2026-09-04T14:46:58.000000","reverse_dns_suffix":"gae.googleusercontent.com (***-***-***-***.gae.googleusercontent.com) or google.com (google-proxy-***-***-***-***.google.com)","verification_method":"Reverse DNS with matching forward DNS, or match the source IP against user-triggered-fetchers.json or user-triggered-fetchers-google.json.","docs_url":"https://developers.google.com/crawling/docs/crawlers-fetchers/overview-google-crawlers","page_url":"https://www.proxies.sx/crawlers/google-publisher-center"}
{"slug":"google-read-aloud","name":"Google Read Aloud","operator":"Google","user_agent_token":"Google-Read-Aloud","purpose_category":"user-triggered-fetch","robots_token":"undocumented","robots_token_is_policy_only":false,"honors_robots_txt":"no","ip_ranges_url":"https://developers.google.com/static/crawling/ipranges/user-triggered-fetchers-google.json","ip_prefix_count":496,"ip_ranges_creation_time":"2026-09-04T14:46:58.000000","reverse_dns_suffix":"gae.googleusercontent.com (***-***-***-***.gae.googleusercontent.com) or google.com (google-proxy-***-***-***-***.google.com)","verification_method":"Reverse DNS with matching forward DNS, or match the source IP against user-triggered-fetchers.json or user-triggered-fetchers-google.json.","docs_url":"https://developers.google.com/crawling/docs/crawlers-fetchers/overview-google-crawlers","page_url":"https://www.proxies.sx/crawlers/google-read-aloud"}
{"slug":"google-site-verifier","name":"Google Site Verifier","operator":"Google","user_agent_token":"Google-Site-Verification","purpose_category":"user-triggered-fetch","robots_token":"undocumented","robots_token_is_policy_only":false,"honors_robots_txt":"no","ip_ranges_url":"https://developers.google.com/static/crawling/ipranges/user-triggered-fetchers-google.json","ip_prefix_count":496,"ip_ranges_creation_time":"2026-09-04T14:46:58.000000","reverse_dns_suffix":"gae.googleusercontent.com (***-***-***-***.gae.googleusercontent.com) or google.com (google-proxy-***-***-***-***.google.com)","verification_method":"Reverse DNS with matching forward DNS, or match the source IP against user-triggered-fetchers.json or user-triggered-fetchers-google.json.","docs_url":"https://developers.google.com/crawling/docs/crawlers-fetchers/overview-google-crawlers","page_url":"https://www.proxies.sx/crawlers/google-site-verifier"}
{"slug":"google-agent","name":"Google-Agent","operator":"Google","user_agent_token":"Google-Agent","purpose_category":"user-triggered-fetch","robots_token":"undocumented","robots_token_is_policy_only":false,"honors_robots_txt":"no","ip_ranges_url":"https://developers.google.com/static/crawling/ipranges/user-triggered-agents.json","ip_prefix_count":20,"ip_ranges_creation_time":"2026-09-03T14:45:47.000000","reverse_dns_suffix":"undocumented for agents specifically; the user-triggered fetchers page gives ***-***-***-***.gae.googleusercontent.com or google-proxy-***-***-***-***.google.com for the category","verification_method":"Match the source IP against user-triggered-agents.json (the docs say 'It uses IP ranges from user-triggered-agents.json'), together with the general reverse and forward DNS check. A subset of Google-Agent requests also carry an experimental Web Bot Auth signature: Signature-Agent: g=\"https://agent.bot.goog\", signed per RFC 9421 HTTP Message Signatures, with public keys at https://agent.bot.goog/.well-known/http-message-signatures-directory. Google says not all requests are signed and to fall back to IP-based verification.","docs_url":"https://developers.google.com/crawling/docs/crawlers-fetchers/overview-google-crawlers","page_url":"https://www.proxies.sx/crawlers/google-agent"}
{"slug":"meta-externalfetcher","name":"Meta-ExternalFetcher","operator":"Meta","user_agent_token":"meta-externalfetcher","purpose_category":"user-triggered-fetch","robots_token":"meta-externalfetcher","robots_token_is_policy_only":false,"honors_robots_txt":"partial","ip_ranges_url":"","ip_prefix_count":"","ip_ranges_creation_time":"","reverse_dns_suffix":"","verification_method":"Meta does not document a verification method for this crawler. The web crawlers page gives only a page-wide statement to allow-list \"the user agent strings or the IP addresses (more secure)\", with no IP file, ASN, or reverse-DNS suffix given.","docs_url":"https://developers.facebook.com/docs/sharing/webmasters/web-crawlers","page_url":"https://www.proxies.sx/crawlers/meta-externalfetcher"}
{"slug":"mistralai-user","name":"MistralAI-User","operator":"Mistral AI","user_agent_token":"MistralAI-User","purpose_category":"user-triggered-fetch","robots_token":"MistralAI-User","robots_token_is_policy_only":false,"honors_robots_txt":"undocumented","ip_ranges_url":"https://mistral.ai/mistralai-user-ips.json","ip_prefix_count":4,"ip_ranges_creation_time":"2025-02-19T12:00:00.000000","reverse_dns_suffix":"","verification_method":"Mistral publishes the IP ranges for this crawler at https://mistral.ai/mistralai-user-ips.json. No reverse-DNS method is documented.","docs_url":"https://docs.mistral.ai/robots","page_url":"https://www.proxies.sx/crawlers/mistralai-user"}
{"slug":"chatgpt-user","name":"ChatGPT-User","operator":"OpenAI","user_agent_token":"ChatGPT-User","purpose_category":"user-triggered-fetch","robots_token":"undocumented","robots_token_is_policy_only":false,"honors_robots_txt":"partial","ip_ranges_url":"https://openai.com/chatgpt-user.json","ip_prefix_count":207,"ip_ranges_creation_time":"2026-09-04T18:03:29.248484","reverse_dns_suffix":"","verification_method":"OpenAI publishes IP ranges for ChatGPT-User at https://openai.com/chatgpt-user.json. No reverse-DNS method is documented.","docs_url":"https://developers.openai.com/api/docs/bots","page_url":"https://www.proxies.sx/crawlers/chatgpt-user"}
{"slug":"perplexity-user","name":"Perplexity-User","operator":"Perplexity","user_agent_token":"Perplexity-User","purpose_category":"user-triggered-fetch","robots_token":"Perplexity-User","robots_token_is_policy_only":false,"honors_robots_txt":"no","ip_ranges_url":"https://www.perplexity.com/perplexity-user.json","ip_prefix_count":4,"ip_ranges_creation_time":"2025-10-17T10:17:00.000000","reverse_dns_suffix":"","verification_method":"The operator publishes IP ranges only. The docs say to combine User-Agent string matching with IP address verification against the published JSON range file. No reverse DNS method is documented.","docs_url":"https://docs.perplexity.ai/guides/bots","page_url":"https://www.proxies.sx/crawlers/perplexity-user"}
{"slug":"yandexcalendar","name":"YandexCalendar","operator":"Yandex","user_agent_token":"YandexCalendar","purpose_category":"user-triggered-fetch","robots_token":"YandexCalendar","robots_token_is_policy_only":false,"honors_robots_txt":"partial","ip_ranges_url":"","ip_prefix_count":"","ip_ranges_creation_time":"","reverse_dns_suffix":"yandex.ru, yandex.net, yandex.com","verification_method":"Reverse DNS lookup on the requesting IP must return a host ending in yandex.ru, yandex.net or yandex.com, and a forward DNS lookup of that host must return the same IP. Yandex Webmaster also provides an IP address check tool.","docs_url":"https://yandex.com/support/webmaster/en/robot-workings/check-yandex-robots.html","page_url":"https://www.proxies.sx/crawlers/yandexcalendar"}
{"slug":"yandexdialogs","name":"YandexDialogs","operator":"Yandex","user_agent_token":"YandexDialogs","purpose_category":"user-triggered-fetch","robots_token":"YandexDialogs","robots_token_is_policy_only":false,"honors_robots_txt":"partial","ip_ranges_url":"","ip_prefix_count":"","ip_ranges_creation_time":"","reverse_dns_suffix":"yandex.ru, yandex.net, yandex.com","verification_method":"Reverse DNS lookup on the requesting IP must return a host ending in yandex.ru, yandex.net or yandex.com, and a forward DNS lookup of that host must return the same IP. Yandex Webmaster also provides an IP address check tool.","docs_url":"https://yandex.com/support/webmaster/en/robot-workings/check-yandex-robots.html","page_url":"https://www.proxies.sx/crawlers/yandexdialogs"}
{"slug":"yandexfeed","name":"YandexFeed","operator":"Yandex","user_agent_token":"YandexFeed","purpose_category":"user-triggered-fetch","robots_token":"YandexFeed","robots_token_is_policy_only":false,"honors_robots_txt":"partial","ip_ranges_url":"","ip_prefix_count":"","ip_ranges_creation_time":"","reverse_dns_suffix":"yandex.ru, yandex.net, yandex.com","verification_method":"Reverse DNS lookup on the requesting IP must return a host ending in yandex.ru, yandex.net or yandex.com, and a forward DNS lookup of that host must return the same IP. Yandex Webmaster also provides an IP address check tool.","docs_url":"https://yandex.com/support/webmaster/en/robot-workings/check-yandex-robots.html","page_url":"https://www.proxies.sx/crawlers/yandexfeed"}
{"slug":"yandexsearchshop","name":"YandexSearchShop","operator":"Yandex","user_agent_token":"YandexSearchShop","purpose_category":"user-triggered-fetch","robots_token":"YandexSearchShop","robots_token_is_policy_only":false,"honors_robots_txt":"partial","ip_ranges_url":"","ip_prefix_count":"","ip_ranges_creation_time":"","reverse_dns_suffix":"yandex.ru, yandex.net, yandex.com","verification_method":"Reverse DNS lookup on the requesting IP must return a host ending in yandex.ru, yandex.net or yandex.com, and a forward DNS lookup of that host must return the same IP. Yandex Webmaster also provides an IP address check tool.","docs_url":"https://yandex.com/support/webmaster/en/robot-workings/check-yandex-robots.html","page_url":"https://www.proxies.sx/crawlers/yandexsearchshop"}
{"slug":"yandexuserproxy","name":"YandexUserproxy","operator":"Yandex","user_agent_token":"YandexUserproxy","purpose_category":"user-triggered-fetch","robots_token":"YandexUserproxy","robots_token_is_policy_only":false,"honors_robots_txt":"partial","ip_ranges_url":"","ip_prefix_count":"","ip_ranges_creation_time":"","reverse_dns_suffix":"yandex.ru, yandex.net, yandex.com","verification_method":"Reverse DNS lookup on the requesting IP must return a host ending in yandex.ru, yandex.net or yandex.com, and a forward DNS lookup of that host must return the same IP. Yandex Webmaster also provides an IP address check tool.","docs_url":"https://yandex.com/support/webmaster/en/robot-workings/check-yandex-robots.html","page_url":"https://www.proxies.sx/crawlers/yandexuserproxy"}
{"slug":"applebot","name":"Applebot","operator":"Apple","user_agent_token":"Applebot","purpose_category":"search-engine","robots_token":"Applebot","robots_token_is_policy_only":false,"honors_robots_txt":"yes","ip_ranges_url":"https://search.developer.apple.com/applebot.json","ip_prefix_count":33,"ip_ranges_creation_time":"2026-07-31T10:00:00.000000","reverse_dns_suffix":".applebot.apple.com","verification_method":"Apple documents two methods. Reverse DNS: the IP should resolve to a host within *.applebot.apple.com, and a forward lookup of that hostname should return the same IP (Apple's example: 17.58.101.179 resolves to 17-58-101-179.applebot.apple.com, which resolves back to 17.58.101.179). IP matching: compare the address against the CIDR prefixes published at https://search.developer.apple.com/applebot.json.","docs_url":"https://support.apple.com/en-us/119829","page_url":"https://www.proxies.sx/crawlers/applebot"}
{"slug":"bytespider","name":"Bytespider","operator":"ByteDance","user_agent_token":"Bytespider","purpose_category":"search-engine","robots_token":"undocumented","robots_token_is_policy_only":false,"honors_robots_txt":"undocumented","ip_ranges_url":"https://zhanzhang.toutiao.com/docs/intro/26899","ip_prefix_count":12,"ip_ranges_creation_time":"","reverse_dns_suffix":".bytedance.com","verification_method":"ByteDance's official documentation says webmasters can verify Bytespider by reverse DNS lookup (host, nslookup -qt=ptr, or dig -x): genuine Bytespider hostnames follow the *.bytedance.com pattern, and anything else is an impostor (冒充). The documentation gives the worked example host 111.225.148.250 -> bytespider-111-225-148-148.crawl.bytedance.com. The same page lists ten IPv4 /24 ranges inline (110.249.201.0/24, 110.249.202.0/24, 111.225.148.0/24, 111.225.149.0/24, 220.243.135.0/24, 220.243.136.0/24, 220.243.188.0/24, 220.243.189.0/24, 60.8.123.0/24, 60.8.151.0/24); no machine-readable JSON or TXT IP range file is published.","docs_url":"https://zhanzhang.toutiao.com/docs/intro/26899","page_url":"https://www.proxies.sx/crawlers/bytespider"}
{"slug":"duckduckbot","name":"DuckDuckBot","operator":"DuckDuckGo","user_agent_token":"DuckDuckBot","purpose_category":"search-engine","robots_token":"undocumented","robots_token_is_policy_only":false,"honors_robots_txt":"yes","ip_ranges_url":"https://duckduckgo.com/duckduckbot.json","ip_prefix_count":486,"ip_ranges_creation_time":"2026-09-01T12:44:58.000000","reverse_dns_suffix":"","verification_method":"Verification is by published IP list only. The help page lists the originating IP addresses inline and links to the machine-readable list at https://duckduckgo.com/duckduckbot.json. The published list contains IPv4 host addresses only, and no reverse-DNS suffix is documented.","docs_url":"https://duckduckgo.com/duckduckgo-help-pages/results/duckduckbot/","page_url":"https://www.proxies.sx/crawlers/duckduckbot"}
{"slug":"google-storebot","name":"Google StoreBot","operator":"Google","user_agent_token":"Storebot-Google","purpose_category":"search-engine","robots_token":"Storebot-Google","robots_token_is_policy_only":false,"honors_robots_txt":"yes","ip_ranges_url":"https://developers.google.com/static/crawling/ipranges/common-crawlers.json","ip_prefix_count":317,"ip_ranges_creation_time":"2026-09-04T14:46:55.000000","reverse_dns_suffix":"googlebot.com (crawl-***-***-***-***.googlebot.com or geo-crawl-***-***-***-***.geo.googlebot.com)","verification_method":"Reverse DNS to googlebot.com, google.com, or googleusercontent.com with matching forward DNS, or match the source IP against the published common-crawlers.json ranges.","docs_url":"https://developers.google.com/crawling/docs/crawlers-fetchers/overview-google-crawlers","page_url":"https://www.proxies.sx/crawlers/google-storebot"}
{"slug":"googlebot","name":"Googlebot","operator":"Google","user_agent_token":"Googlebot","purpose_category":"search-engine","robots_token":"Googlebot","robots_token_is_policy_only":false,"honors_robots_txt":"yes","ip_ranges_url":"https://developers.google.com/static/crawling/ipranges/common-crawlers.json","ip_prefix_count":317,"ip_ranges_creation_time":"2026-09-04T14:46:55.000000","reverse_dns_suffix":"googlebot.com (crawl-***-***-***-***.googlebot.com or geo-crawl-***-***-***-***.geo.googlebot.com)","verification_method":"A reverse DNS lookup of the source IP must resolve to googlebot.com, google.com, or googleusercontent.com, and a forward DNS lookup of that name must return the same IP. Alternatively, match the IP against the published common-crawlers.json CIDR ranges.","docs_url":"https://developers.google.com/crawling/docs/crawlers-fetchers/overview-google-crawlers","page_url":"https://www.proxies.sx/crawlers/googlebot"}
{"slug":"googlebot-image","name":"Googlebot Image","operator":"Google","user_agent_token":"Googlebot-Image","purpose_category":"search-engine","robots_token":"Googlebot-Image","robots_token_is_policy_only":false,"honors_robots_txt":"yes","ip_ranges_url":"https://developers.google.com/static/crawling/ipranges/common-crawlers.json","ip_prefix_count":317,"ip_ranges_creation_time":"2026-09-04T14:46:55.000000","reverse_dns_suffix":"googlebot.com (crawl-***-***-***-***.googlebot.com or geo-crawl-***-***-***-***.geo.googlebot.com)","verification_method":"Reverse DNS to googlebot.com, google.com, or googleusercontent.com with matching forward DNS, or match the source IP against the published common-crawlers.json ranges.","docs_url":"https://developers.google.com/crawling/docs/crawlers-fetchers/overview-google-crawlers","page_url":"https://www.proxies.sx/crawlers/googlebot-image"}
{"slug":"googlebot-news","name":"Googlebot News","operator":"Google","user_agent_token":"Googlebot-News","purpose_category":"search-engine","robots_token":"Googlebot-News","robots_token_is_policy_only":true,"honors_robots_txt":"yes","ip_ranges_url":"https://developers.google.com/static/crawling/ipranges/common-crawlers.json","ip_prefix_count":317,"ip_ranges_creation_time":"2026-09-04T14:46:55.000000","reverse_dns_suffix":"googlebot.com (crawl-***-***-***-***.googlebot.com or geo-crawl-***-***-***-***.geo.googlebot.com)","verification_method":"Reverse DNS to googlebot.com, google.com, or googleusercontent.com with matching forward DNS, or match the source IP against the published common-crawlers.json ranges.","docs_url":"https://developers.google.com/crawling/docs/crawlers-fetchers/overview-google-crawlers","page_url":"https://www.proxies.sx/crawlers/googlebot-news"}
{"slug":"googlebot-video","name":"Googlebot Video","operator":"Google","user_agent_token":"Googlebot-Video","purpose_category":"search-engine","robots_token":"Googlebot-Video","robots_token_is_policy_only":false,"honors_robots_txt":"yes","ip_ranges_url":"https://developers.google.com/static/crawling/ipranges/common-crawlers.json","ip_prefix_count":317,"ip_ranges_creation_time":"2026-09-04T14:46:55.000000","reverse_dns_suffix":"googlebot.com (crawl-***-***-***-***.googlebot.com or geo-crawl-***-***-***-***.geo.googlebot.com)","verification_method":"Reverse DNS to googlebot.com, google.com, or googleusercontent.com with matching forward DNS, or match the source IP against the published common-crawlers.json ranges.","docs_url":"https://developers.google.com/crawling/docs/crawlers-fetchers/overview-google-crawlers","page_url":"https://www.proxies.sx/crawlers/googlebot-video"}
{"slug":"bingvideopreview","name":"BingVideoPreview","operator":"Microsoft Bing","user_agent_token":"BingVideoPreview/1.0","purpose_category":"search-engine","robots_token":"undocumented","robots_token_is_policy_only":false,"honors_robots_txt":"undocumented","ip_ranges_url":"","ip_prefix_count":"","ip_ranges_creation_time":"","reverse_dns_suffix":"","verification_method":"Microsoft does not document a verification method specific to BingVideoPreview. The generic guidance for all Bing crawlers applies: use the Verify Bingbot tool, check that reverse DNS resolves to a hostname under search.msn.com, or match the IP against the published list at https://www.bing.com/toolbox/bingbot.json.","docs_url":"https://www.bing.com/webmasters/help/which-crawlers-does-bing-use-8c184ec0","page_url":"https://www.proxies.sx/crawlers/bingvideopreview"}
{"slug":"bingbot","name":"Bingbot","operator":"Microsoft Bing","user_agent_token":"bingbot/2.0","purpose_category":"search-engine","robots_token":"Bingbot","robots_token_is_policy_only":false,"honors_robots_txt":"yes","ip_ranges_url":"https://www.bing.com/toolbox/bingbot.json","ip_prefix_count":28,"ip_ranges_creation_time":"2024-01-03T10:00:00.121331","reverse_dns_suffix":"search.msn.com","verification_method":"A reverse DNS lookup of the IP must resolve to a hostname ending in search.msn.com, and a forward DNS lookup of that hostname must return the same IP (the documentation's example hostname is msnbot-157-55-33-18.search.msn.com). Alternatively, match the IP against the published list at https://www.bing.com/toolbox/bingbot.json (\"Refresh this list daily as we can change this list any time\"), or use the Verify Bingbot tool, available publicly at http://www.bing.com/toolbox/verify-bingbot or inside Bing Webmaster Tools under Tools & Enhancements. The documentation carries the heading \"Don't Use Hardcoded IP Addresses or Address Ranges\".","docs_url":"https://www.bing.com/webmasters/help/which-crawlers-does-bing-use-8c184ec0","page_url":"https://www.proxies.sx/crawlers/bingbot"}
{"slug":"yacombot","name":"YaComBot","operator":"Yandex","user_agent_token":"YaComBot","purpose_category":"search-engine","robots_token":"YaComBot","robots_token_is_policy_only":false,"honors_robots_txt":"yes","ip_ranges_url":"","ip_prefix_count":"","ip_ranges_creation_time":"","reverse_dns_suffix":"yandex.ru, yandex.net, yandex.com","verification_method":"Reverse DNS lookup on the requesting IP must return a host ending in yandex.ru, yandex.net or yandex.com, and a forward DNS lookup of that host must return the same IP. Yandex Webmaster also provides an IP address check tool.","docs_url":"https://yandex.com/support/webmaster/en/robot-workings/check-yandex-robots.html","page_url":"https://www.proxies.sx/crawlers/yacombot"}
{"slug":"yandexblogs","name":"YandexBlogs","operator":"Yandex","user_agent_token":"YandexBlogs","purpose_category":"search-engine","robots_token":"YandexBlogs","robots_token_is_policy_only":false,"honors_robots_txt":"yes","ip_ranges_url":"","ip_prefix_count":"","ip_ranges_creation_time":"","reverse_dns_suffix":"yandex.ru, yandex.net, yandex.com","verification_method":"Reverse DNS lookup on the requesting IP must return a host ending in yandex.ru, yandex.net or yandex.com, and a forward DNS lookup of that host must return the same IP. Yandex Webmaster also provides an IP address check tool.","docs_url":"https://yandex.com/support/webmaster/en/robot-workings/check-yandex-robots.html","page_url":"https://www.proxies.sx/crawlers/yandexblogs"}
{"slug":"yandexbot","name":"YandexBot","operator":"Yandex","user_agent_token":"YandexBot","purpose_category":"search-engine","robots_token":"YandexBot","robots_token_is_policy_only":false,"honors_robots_txt":"yes","ip_ranges_url":"","ip_prefix_count":"","ip_ranges_creation_time":"","reverse_dns_suffix":"yandex.ru, yandex.net, yandex.com","verification_method":"Reverse DNS lookup on the requesting IP must return a host ending in yandex.ru, yandex.net or yandex.com, and a forward DNS lookup of that host must return the same IP. Yandex Webmaster also offers an 'IP address check' tool (Tools -> IP address check). Yandex bots use AS13238, AS208722 and AS212066 with frequently changing IPs, and the IP list is not disclosed.","docs_url":"https://yandex.com/support/webmaster/en/robot-workings/check-yandex-robots.html","page_url":"https://www.proxies.sx/crawlers/yandexbot"}
{"slug":"yandexcombot","name":"YandexComBot","operator":"Yandex","user_agent_token":"YandexComBot","purpose_category":"search-engine","robots_token":"YandexComBot","robots_token_is_policy_only":false,"honors_robots_txt":"partial","ip_ranges_url":"","ip_prefix_count":"","ip_ranges_creation_time":"","reverse_dns_suffix":"yandex.ru, yandex.net, yandex.com","verification_method":"Reverse DNS lookup on the requesting IP must return a host ending in yandex.ru, yandex.net or yandex.com, and a forward DNS lookup of that host must return the same IP. Yandex Webmaster also provides an IP address check tool.","docs_url":"https://yandex.com/support/webmaster/en/robot-workings/check-yandex-robots.html","page_url":"https://www.proxies.sx/crawlers/yandexcombot"}
{"slug":"yandexfavicons","name":"YandexFavicons","operator":"Yandex","user_agent_token":"YandexFavicons","purpose_category":"search-engine","robots_token":"YandexFavicons","robots_token_is_policy_only":false,"honors_robots_txt":"partial","ip_ranges_url":"","ip_prefix_count":"","ip_ranges_creation_time":"","reverse_dns_suffix":"yandex.ru, yandex.net, yandex.com","verification_method":"Reverse DNS lookup on the requesting IP must return a host ending in yandex.ru, yandex.net or yandex.com, and a forward DNS lookup of that host must return the same IP. Yandex Webmaster also provides an IP address check tool.","docs_url":"https://yandex.com/support/webmaster/en/robot-workings/check-yandex-robots.html","page_url":"https://www.proxies.sx/crawlers/yandexfavicons"}
{"slug":"yandeximages","name":"YandexImages","operator":"Yandex","user_agent_token":"YandexImages","purpose_category":"search-engine","robots_token":"YandexImages","robots_token_is_policy_only":false,"honors_robots_txt":"yes","ip_ranges_url":"","ip_prefix_count":"","ip_ranges_creation_time":"","reverse_dns_suffix":"yandex.ru, yandex.net, yandex.com","verification_method":"Reverse DNS lookup on the requesting IP must return a host ending in yandex.ru, yandex.net or yandex.com, and a forward DNS lookup of that host must return the same IP. Yandex Webmaster also provides an IP address check tool.","docs_url":"https://yandex.com/support/webmaster/en/robot-workings/check-yandex-robots.html","page_url":"https://www.proxies.sx/crawlers/yandeximages"}
{"slug":"yandexmedia","name":"YandexMedia","operator":"Yandex","user_agent_token":"YandexMedia","purpose_category":"search-engine","robots_token":"YandexMedia","robots_token_is_policy_only":false,"honors_robots_txt":"yes","ip_ranges_url":"","ip_prefix_count":"","ip_ranges_creation_time":"","reverse_dns_suffix":"yandex.ru, yandex.net, yandex.com","verification_method":"Reverse DNS lookup on the requesting IP must return a host ending in yandex.ru, yandex.net or yandex.com, and a forward DNS lookup of that host must return the same IP. Yandex Webmaster also provides an IP address check tool.","docs_url":"https://yandex.com/support/webmaster/en/robot-workings/check-yandex-robots.html","page_url":"https://www.proxies.sx/crawlers/yandexmedia"}
{"slug":"yandexmobilebot","name":"YandexMobileBot","operator":"Yandex","user_agent_token":"YandexMobileBot","purpose_category":"search-engine","robots_token":"YandexMobileBot","robots_token_is_policy_only":false,"honors_robots_txt":"partial","ip_ranges_url":"","ip_prefix_count":"","ip_ranges_creation_time":"","reverse_dns_suffix":"yandex.ru, yandex.net, yandex.com","verification_method":"Reverse DNS lookup on the requesting IP must return a host ending in yandex.ru, yandex.net or yandex.com, and a forward DNS lookup of that host must return the same IP. Yandex Webmaster also provides an IP address check tool.","docs_url":"https://yandex.com/support/webmaster/en/robot-workings/check-yandex-robots.html","page_url":"https://www.proxies.sx/crawlers/yandexmobilebot"}
{"slug":"yandexontodb","name":"YandexOntoDB","operator":"Yandex","user_agent_token":"YandexOntoDB","purpose_category":"search-engine","robots_token":"YandexOntoDB","robots_token_is_policy_only":false,"honors_robots_txt":"yes","ip_ranges_url":"","ip_prefix_count":"","ip_ranges_creation_time":"","reverse_dns_suffix":"yandex.ru, yandex.net, yandex.com","verification_method":"Reverse DNS lookup on the requesting IP must return a host ending in yandex.ru, yandex.net or yandex.com, and a forward DNS lookup of that host must return the same IP. Yandex Webmaster also provides an IP address check tool.","docs_url":"https://yandex.com/support/webmaster/en/robot-workings/check-yandex-robots.html","page_url":"https://www.proxies.sx/crawlers/yandexontodb"}
{"slug":"yandexrenderresourcesbot","name":"YandexRenderResourcesBot","operator":"Yandex","user_agent_token":"YandexRenderResourcesBot","purpose_category":"search-engine","robots_token":"YandexRenderResourcesBot","robots_token_is_policy_only":false,"honors_robots_txt":"partial","ip_ranges_url":"","ip_prefix_count":"","ip_ranges_creation_time":"","reverse_dns_suffix":"yandex.ru, yandex.net, yandex.com","verification_method":"Reverse DNS lookup on the requesting IP must return a host ending in yandex.ru, yandex.net or yandex.com, and a forward DNS lookup of that host must return the same IP. Yandex Webmaster also provides an IP address check tool.","docs_url":"https://yandex.com/support/webmaster/en/robot-workings/check-yandex-robots.html","page_url":"https://www.proxies.sx/crawlers/yandexrenderresourcesbot"}
{"slug":"yandexsitelinks","name":"YandexSitelinks","operator":"Yandex","user_agent_token":"YandexSitelinks","purpose_category":"search-engine","robots_token":"YandexSitelinks","robots_token_is_policy_only":false,"honors_robots_txt":"yes","ip_ranges_url":"","ip_prefix_count":"","ip_ranges_creation_time":"","reverse_dns_suffix":"yandex.ru, yandex.net, yandex.com","verification_method":"Reverse DNS lookup on the requesting IP must return a host ending in yandex.ru, yandex.net or yandex.com, and a forward DNS lookup of that host must return the same IP. Yandex Webmaster also provides an IP address check tool.","docs_url":"https://yandex.com/support/webmaster/en/robot-workings/check-yandex-robots.html","page_url":"https://www.proxies.sx/crawlers/yandexsitelinks"}
{"slug":"yandexverticals","name":"YandexVerticals","operator":"Yandex","user_agent_token":"YandexVerticals","purpose_category":"search-engine","robots_token":"YandexVerticals","robots_token_is_policy_only":false,"honors_robots_txt":"yes","ip_ranges_url":"","ip_prefix_count":"","ip_ranges_creation_time":"","reverse_dns_suffix":"yandex.ru, yandex.net, yandex.com","verification_method":"Reverse DNS lookup on the requesting IP must return a host ending in yandex.ru, yandex.net or yandex.com, and a forward DNS lookup of that host must return the same IP. Yandex Webmaster also provides an IP address check tool.","docs_url":"https://yandex.com/support/webmaster/en/robot-workings/check-yandex-robots.html","page_url":"https://www.proxies.sx/crawlers/yandexverticals"}
{"slug":"yandexvertis","name":"YandexVertis","operator":"Yandex","user_agent_token":"YandexVertis","purpose_category":"search-engine","robots_token":"YandexVertis","robots_token_is_policy_only":false,"honors_robots_txt":"yes","ip_ranges_url":"","ip_prefix_count":"","ip_ranges_creation_time":"","reverse_dns_suffix":"yandex.ru, yandex.net, yandex.com","verification_method":"Reverse DNS lookup on the requesting IP must return a host ending in yandex.ru, yandex.net or yandex.com, and a forward DNS lookup of that host must return the same IP. Yandex Webmaster also provides an IP address check tool.","docs_url":"https://yandex.com/support/webmaster/en/robot-workings/check-yandex-robots.html","page_url":"https://www.proxies.sx/crawlers/yandexvertis"}
{"slug":"yandexvideo","name":"YandexVideo","operator":"Yandex","user_agent_token":"YandexVideo","purpose_category":"search-engine","robots_token":"YandexVideo","robots_token_is_policy_only":false,"honors_robots_txt":"yes","ip_ranges_url":"","ip_prefix_count":"","ip_ranges_creation_time":"","reverse_dns_suffix":"yandex.ru, yandex.net, yandex.com","verification_method":"Reverse DNS lookup on the requesting IP must return a host ending in yandex.ru, yandex.net or yandex.com, and a forward DNS lookup of that host must return the same IP. Yandex Webmaster also provides an IP address check tool.","docs_url":"https://yandex.com/support/webmaster/en/robot-workings/check-yandex-robots.html","page_url":"https://www.proxies.sx/crawlers/yandexvideo"}
{"slug":"yandexvideoparser","name":"YandexVideoParser","operator":"Yandex","user_agent_token":"YandexVideoParser","purpose_category":"search-engine","robots_token":"YandexVideoParser","robots_token_is_policy_only":false,"honors_robots_txt":"partial","ip_ranges_url":"","ip_prefix_count":"","ip_ranges_creation_time":"","reverse_dns_suffix":"yandex.ru, yandex.net, yandex.com","verification_method":"Reverse DNS lookup on the requesting IP must return a host ending in yandex.ru, yandex.net or yandex.com, and a forward DNS lookup of that host must return the same IP. Yandex Webmaster also provides an IP address check tool.","docs_url":"https://yandex.com/support/webmaster/en/robot-workings/check-yandex-robots.html","page_url":"https://www.proxies.sx/crawlers/yandexvideoparser"}
{"slug":"google-messages","name":"Google Messages","operator":"Google","user_agent_token":"GoogleMessages","purpose_category":"social-preview","robots_token":"undocumented","robots_token_is_policy_only":false,"honors_robots_txt":"no","ip_ranges_url":"https://developers.google.com/static/crawling/ipranges/user-triggered-fetchers-google.json","ip_prefix_count":496,"ip_ranges_creation_time":"2026-09-04T14:46:58.000000","reverse_dns_suffix":"gae.googleusercontent.com (***-***-***-***.gae.googleusercontent.com) or google.com (google-proxy-***-***-***-***.google.com)","verification_method":"Reverse DNS with matching forward DNS, or match the source IP against user-triggered-fetchers.json or user-triggered-fetchers-google.json.","docs_url":"https://developers.google.com/crawling/docs/crawlers-fetchers/overview-google-crawlers","page_url":"https://www.proxies.sx/crawlers/google-messages"}
{"slug":"facebookexternalhit","name":"FacebookExternalHit","operator":"Meta","user_agent_token":"facebookexternalhit","purpose_category":"social-preview","robots_token":"facebookexternalhit","robots_token_is_policy_only":false,"honors_robots_txt":"partial","ip_ranges_url":"","ip_prefix_count":"","ip_ranges_creation_time":"","reverse_dns_suffix":"","verification_method":"Meta's current crawler page gives no method beyond \"Add to your allow list either the user agent strings or the IP addresses (more secure) used by the crawler.\" The retired Facebook Crawler page documented: \"To get a current list of IP addresses the crawler uses, run the following command. whois -h whois.radb.net -- '-i origin AS32934' | grep ^route\" and \"These IP addresses change often.\" Meta currently publishes a variant of that command on its WhatsApp webhooks page, described as returning \"the IP addresses of Meta's webhook servers\" rather than crawlers: \"whois -h whois.radb.net -- '-i origin AS32934' | grep '^route' | awk '{print $2}' | sort\", and links a geofeed CSV at https://facebook.com/peering/geofeed (\"download a CSV that lists Meta's IP addresses\"); no reverse-DNS method is documented.","docs_url":"https://developers.facebook.com/docs/sharing/webmasters/web-crawlers","page_url":"https://www.proxies.sx/crawlers/facebookexternalhit"}
{"slug":"microsoftpreview","name":"MicrosoftPreview","operator":"Microsoft Bing","user_agent_token":"MicrosoftPreview/2.0","purpose_category":"social-preview","robots_token":"undocumented","robots_token_is_policy_only":false,"honors_robots_txt":"undocumented","ip_ranges_url":"","ip_prefix_count":"","ip_ranges_creation_time":"","reverse_dns_suffix":"","verification_method":"Microsoft does not document a verification method specific to MicrosoftPreview. The generic guidance for all Bing crawlers applies: use the Verify Bingbot tool, check that reverse DNS resolves to a hostname under search.msn.com, or match the IP against the published list at https://www.bing.com/toolbox/bingbot.json.","docs_url":"https://www.bing.com/webmasters/help/which-crawlers-does-bing-use-8c184ec0","page_url":"https://www.proxies.sx/crawlers/microsoftpreview"}
{"slug":"ahrefsbot","name":"AhrefsBot","operator":"Ahrefs Pte. Ltd.","user_agent_token":"AhrefsBot","purpose_category":"seo-tool","robots_token":"AhrefsBot","robots_token_is_policy_only":false,"honors_robots_txt":"yes","ip_ranges_url":"https://api.ahrefs.com/v3/public/crawler-ip-ranges","ip_prefix_count":81,"ip_ranges_creation_time":"","reverse_dns_suffix":"ahrefs.com or ahrefs.net","verification_method":"Ahrefs publishes its crawler IP ranges at https://api.ahrefs.com/v3/public/crawler-ip-ranges and individual crawler IP addresses at https://api.ahrefs.com/v3/public/crawler-ips. Reverse DNS can also be used: 'The reverse DNS suffix of the IPs hostname is always ahrefs.com or ahrefs.net.' AhrefsBot is also listed as a Cloudflare verified 'good' bot.","docs_url":"https://ahrefs.com/robot","page_url":"https://www.proxies.sx/crawlers/ahrefsbot"}
{"slug":"ahrefssiteaudit","name":"AhrefsSiteAudit","operator":"Ahrefs Pte. Ltd.","user_agent_token":"AhrefsSiteAudit","purpose_category":"seo-tool","robots_token":"AhrefsSiteAudit","robots_token_is_policy_only":false,"honors_robots_txt":"partial","ip_ranges_url":"https://api.ahrefs.com/v3/public/crawler-ip-ranges","ip_prefix_count":81,"ip_ranges_creation_time":"","reverse_dns_suffix":"ahrefs.com or ahrefs.net","verification_method":"AhrefsSiteAudit uses the same published IP lists as AhrefsBot: https://api.ahrefs.com/v3/public/crawler-ip-ranges and https://api.ahrefs.com/v3/public/crawler-ips. The reverse DNS suffix of its hostnames is 'always ahrefs.com or ahrefs.net'. It is also listed as a Cloudflare verified 'good' bot.","docs_url":"https://ahrefs.com/robot","page_url":"https://www.proxies.sx/crawlers/ahrefssiteaudit"}
{"slug":"google-inspectiontool","name":"Google-InspectionTool","operator":"Google","user_agent_token":"Google-InspectionTool","purpose_category":"seo-tool","robots_token":"Google-InspectionTool","robots_token_is_policy_only":false,"honors_robots_txt":"yes","ip_ranges_url":"https://developers.google.com/static/crawling/ipranges/common-crawlers.json","ip_prefix_count":317,"ip_ranges_creation_time":"2026-09-04T14:46:55.000000","reverse_dns_suffix":"googlebot.com (crawl-***-***-***-***.googlebot.com or geo-crawl-***-***-***-***.geo.googlebot.com)","verification_method":"Reverse DNS to googlebot.com, google.com, or googleusercontent.com with matching forward DNS, or match the source IP against the published common-crawlers.json ranges.","docs_url":"https://developers.google.com/crawling/docs/crawlers-fetchers/overview-google-crawlers","page_url":"https://www.proxies.sx/crawlers/google-inspectiontool"}
{"slug":"rytebot","name":"RyteBot","operator":"Semrush","user_agent_token":"RyteBot","purpose_category":"seo-tool","robots_token":"RyteBot","robots_token_is_policy_only":false,"honors_robots_txt":"yes","ip_ranges_url":"","ip_prefix_count":"","ip_ranges_creation_time":"","reverse_dns_suffix":"","verification_method":"The operator does not document a verification method.","docs_url":"https://www.semrush.com/bot/","page_url":"https://www.proxies.sx/crawlers/rytebot"}
{"slug":"semrushbot","name":"SemrushBot","operator":"Semrush","user_agent_token":"SemrushBot","purpose_category":"seo-tool","robots_token":"SemrushBot","robots_token_is_policy_only":false,"honors_robots_txt":"yes","ip_ranges_url":"","ip_prefix_count":"","ip_ranges_creation_time":"","reverse_dns_suffix":"","verification_method":"The operator does not document a verification method. Semrush states: \"Do not try to block SemrushBot via IP as we do not use any consecutive IP blocks.\" No reverse-DNS suffix and no IP range file are published.","docs_url":"https://www.semrush.com/bot/","page_url":"https://www.proxies.sx/crawlers/semrushbot"}
{"slug":"semrushbot-ba","name":"SemrushBot-BA","operator":"Semrush","user_agent_token":"SemrushBot-BA","purpose_category":"seo-tool","robots_token":"SemrushBot-BA","robots_token_is_policy_only":false,"honors_robots_txt":"yes","ip_ranges_url":"","ip_prefix_count":"","ip_ranges_creation_time":"","reverse_dns_suffix":"","verification_method":"The operator does not document a verification method. Semrush states that \"we do not use any consecutive IP blocks\".","docs_url":"https://www.semrush.com/bot/","page_url":"https://www.proxies.sx/crawlers/semrushbot-ba"}
{"slug":"semrushbot-esi","name":"SemrushBot-ESI","operator":"Semrush","user_agent_token":"SemrushBot-ESI","purpose_category":"seo-tool","robots_token":"SemrushBot-ESI","robots_token_is_policy_only":false,"honors_robots_txt":"yes","ip_ranges_url":"","ip_prefix_count":"","ip_ranges_creation_time":"","reverse_dns_suffix":"","verification_method":"The operator does not document a verification method. Semrush states that \"we do not use any consecutive IP blocks\".","docs_url":"https://www.semrush.com/bot/","page_url":"https://www.proxies.sx/crawlers/semrushbot-esi"}
{"slug":"semrushbot-ft","name":"SemrushBot-FT","operator":"Semrush","user_agent_token":"SemrushBot-FT","purpose_category":"seo-tool","robots_token":"SemrushBot-FT","robots_token_is_policy_only":false,"honors_robots_txt":"yes","ip_ranges_url":"","ip_prefix_count":"","ip_ranges_creation_time":"","reverse_dns_suffix":"","verification_method":"The operator does not document a verification method. Semrush states that \"we do not use any consecutive IP blocks\".","docs_url":"https://www.semrush.com/bot/","page_url":"https://www.proxies.sx/crawlers/semrushbot-ft"}
{"slug":"semrushbot-ocob","name":"SemrushBot-OCOB","operator":"Semrush","user_agent_token":"SemrushBot-OCOB","purpose_category":"seo-tool","robots_token":"SemrushBot-OCOB","robots_token_is_policy_only":false,"honors_robots_txt":"yes","ip_ranges_url":"","ip_prefix_count":"","ip_ranges_creation_time":"","reverse_dns_suffix":"","verification_method":"The operator does not document a verification method. Semrush states that \"we do not use any consecutive IP blocks\".","docs_url":"https://www.semrush.com/bot/","page_url":"https://www.proxies.sx/crawlers/semrushbot-ocob"}
{"slug":"semrushbot-si","name":"SemrushBot-SI","operator":"Semrush","user_agent_token":"SemrushBot-SI","purpose_category":"seo-tool","robots_token":"SemrushBot-SI","robots_token_is_policy_only":false,"honors_robots_txt":"yes","ip_ranges_url":"","ip_prefix_count":"","ip_ranges_creation_time":"","reverse_dns_suffix":"","verification_method":"Semrush publishes IP addresses in Knowledge Base prose rather than a machine-readable file: article 375 says to \"whitelist the following IP addresses and User-agent\", listing \"85.208.98.53\", \"85.208.98.0/24\" and \"User-agent: SemrushBot-SI\", on \"Port 80: HTTP\" and \"Port 443: HTTPS\". No reverse-DNS suffix is documented.","docs_url":"https://www.semrush.com/bot/","page_url":"https://www.proxies.sx/crawlers/semrushbot-si"}
{"slug":"semrushbot-swa","name":"SemrushBot-SWA","operator":"Semrush","user_agent_token":"SemrushBot-SWA","purpose_category":"seo-tool","robots_token":"SemrushBot-SWA","robots_token_is_policy_only":false,"honors_robots_txt":"yes","ip_ranges_url":"","ip_prefix_count":"","ip_ranges_creation_time":"","reverse_dns_suffix":"","verification_method":"The operator does not document a verification method. Semrush states that \"we do not use any consecutive IP blocks\".","docs_url":"https://www.semrush.com/bot/","page_url":"https://www.proxies.sx/crawlers/semrushbot-swa"}
{"slug":"siteauditbot","name":"SiteAuditBot","operator":"Semrush","user_agent_token":"SiteAuditBot","purpose_category":"seo-tool","robots_token":"SiteAuditBot","robots_token_is_policy_only":false,"honors_robots_txt":"partial","ip_ranges_url":"","ip_prefix_count":"","ip_ranges_creation_time":"","reverse_dns_suffix":"","verification_method":"Semrush publishes an IP range in Knowledge Base prose rather than a machine-readable file: \"85.208.98.128/25 (a subnet used by Site Audit only)\". \"The bot is using standard 80 HTTP and 443 HTTPS ports to connect.\" No reverse-DNS suffix is documented.","docs_url":"https://www.semrush.com/bot/","page_url":"https://www.proxies.sx/crawlers/siteauditbot"}
{"slug":"splitsignalbot","name":"SplitSignalBot","operator":"Semrush","user_agent_token":"SplitSignalBot","purpose_category":"seo-tool","robots_token":"SplitSignalBot","robots_token_is_policy_only":false,"honors_robots_txt":"yes","ip_ranges_url":"","ip_prefix_count":"","ip_ranges_creation_time":"","reverse_dns_suffix":"","verification_method":"The operator does not document a verification method. Semrush states that \"we do not use any consecutive IP blocks\".","docs_url":"https://www.semrush.com/bot/","page_url":"https://www.proxies.sx/crawlers/splitsignalbot"}
{"slug":"yandexpagechecker","name":"YandexPagechecker","operator":"Yandex","user_agent_token":"YandexPagechecker","purpose_category":"seo-tool","robots_token":"YandexPagechecker","robots_token_is_policy_only":false,"honors_robots_txt":"yes","ip_ranges_url":"","ip_prefix_count":"","ip_ranges_creation_time":"","reverse_dns_suffix":"yandex.ru, yandex.net, yandex.com","verification_method":"Reverse DNS lookup on the requesting IP must return a host ending in yandex.ru, yandex.net or yandex.com, and a forward DNS lookup of that host must return the same IP. Yandex Webmaster also provides an IP address check tool.","docs_url":"https://yandex.com/support/webmaster/en/robot-workings/check-yandex-robots.html","page_url":"https://www.proxies.sx/crawlers/yandexpagechecker"}
{"slug":"yandexwebmaster","name":"YandexWebmaster","operator":"Yandex","user_agent_token":"YandexWebmaster","purpose_category":"seo-tool","robots_token":"YandexWebmaster","robots_token_is_policy_only":false,"honors_robots_txt":"yes","ip_ranges_url":"","ip_prefix_count":"","ip_ranges_creation_time":"","reverse_dns_suffix":"yandex.ru, yandex.net, yandex.com","verification_method":"Reverse DNS lookup on the requesting IP must return a host ending in yandex.ru, yandex.net or yandex.com, and a forward DNS lookup of that host must return the same IP. Yandex Webmaster also provides an IP address check tool.","docs_url":"https://yandex.com/support/webmaster/en/robot-workings/check-yandex-robots.html","page_url":"https://www.proxies.sx/crawlers/yandexwebmaster"}
{"slug":"ccbot","name":"CCBot","operator":"Common Crawl","user_agent_token":"CCBot","purpose_category":"archive","robots_token":"CCBot","robots_token_is_policy_only":false,"honors_robots_txt":"yes","ip_ranges_url":"https://index.commoncrawl.org/ccbot.json","ip_prefix_count":5,"ip_ranges_creation_time":"2026-08-11T13:40:00Z","reverse_dns_suffix":".crawl.commoncrawl.org","verification_method":"Common Crawl publishes its IP ranges at https://index.commoncrawl.org/ccbot.json and lists the same CIDRs inline in its FAQ, where the IPv6 range is labelled \"IPv6 (no reverse DNS)\"; the file's notes read: \"IP ranges used by CCBot. For verification of IPv4 addresses, FCrDNS is also recommended.\" The operator states: \"CCBot is now run on dedicated IP address ranges with reverse DNS (except over IPv6 where reverse DNS is not yet supported.)\" and gives the example that 18.97.14.84 resolves to 18-97-14-84.crawl.commoncrawl.org, which resolves back to 18.97.14.84. The documentation warns: \"we are aware of crawlers falsely identifying themselves as CCBot. We recommend verifying UserAgent strings to ensure authenticity.\"","docs_url":"https://commoncrawl.org/ccbot","page_url":"https://www.proxies.sx/crawlers/ccbot"}
{"slug":"itms","name":"iTMS","operator":"Apple","user_agent_token":"iTMS","purpose_category":"other","robots_token":"none (does not follow robots.txt)","robots_token_is_policy_only":false,"honors_robots_txt":"no","ip_ranges_url":"","ip_prefix_count":"","ip_ranges_creation_time":"","reverse_dns_suffix":".applebot.apple.com","verification_method":"Apple says iTMS traffic 'may come from applebot.apple.com hosts', so the Applebot reverse-DNS check applies to that traffic. No dedicated IP range file is documented for iTMS.","docs_url":"https://support.apple.com/en-us/119829","page_url":"https://www.proxies.sx/crawlers/itms"}
{"slug":"apis-google","name":"APIs-Google","operator":"Google","user_agent_token":"APIs-Google","purpose_category":"other","robots_token":"APIs-Google","robots_token_is_policy_only":false,"honors_robots_txt":"partial","ip_ranges_url":"https://developers.google.com/static/crawling/ipranges/special-crawlers.json","ip_prefix_count":272,"ip_ranges_creation_time":"2026-09-04T14:45:59.000000","reverse_dns_suffix":"google.com (rate-limited-proxy-***-***-***-***.google.com)","verification_method":"Reverse DNS to rate-limited-proxy-***-***-***-***.google.com with matching forward DNS, or match the source IP against special-crawlers.json.","docs_url":"https://developers.google.com/crawling/docs/crawlers-fetchers/overview-google-crawlers","page_url":"https://www.proxies.sx/crawlers/apis-google"}
{"slug":"adsense","name":"AdSense","operator":"Google","user_agent_token":"Mediapartners-Google","purpose_category":"other","robots_token":"Mediapartners-Google","robots_token_is_policy_only":false,"honors_robots_txt":"partial","ip_ranges_url":"https://developers.google.com/static/crawling/ipranges/special-crawlers.json","ip_prefix_count":272,"ip_ranges_creation_time":"2026-09-04T14:45:59.000000","reverse_dns_suffix":"google.com (rate-limited-proxy-***-***-***-***.google.com)","verification_method":"Reverse DNS to rate-limited-proxy-***-***-***-***.google.com with matching forward DNS, or match the source IP against special-crawlers.json.","docs_url":"https://developers.google.com/crawling/docs/crawlers-fetchers/overview-google-crawlers","page_url":"https://www.proxies.sx/crawlers/adsense"}
{"slug":"adsbot","name":"AdsBot","operator":"Google","user_agent_token":"AdsBot-Google","purpose_category":"other","robots_token":"AdsBot-Google","robots_token_is_policy_only":false,"honors_robots_txt":"partial","ip_ranges_url":"https://developers.google.com/static/crawling/ipranges/special-crawlers.json","ip_prefix_count":272,"ip_ranges_creation_time":"2026-09-04T14:45:59.000000","reverse_dns_suffix":"google.com (rate-limited-proxy-***-***-***-***.google.com)","verification_method":"Reverse DNS to rate-limited-proxy-***-***-***-***.google.com with matching forward DNS, or match the source IP against special-crawlers.json.","docs_url":"https://developers.google.com/crawling/docs/crawlers-fetchers/overview-google-crawlers","page_url":"https://www.proxies.sx/crawlers/adsbot"}
{"slug":"adsbot-mobile-web","name":"AdsBot Mobile Web","operator":"Google","user_agent_token":"AdsBot-Google-Mobile","purpose_category":"other","robots_token":"AdsBot-Google-Mobile","robots_token_is_policy_only":false,"honors_robots_txt":"partial","ip_ranges_url":"https://developers.google.com/static/crawling/ipranges/special-crawlers.json","ip_prefix_count":272,"ip_ranges_creation_time":"2026-09-04T14:45:59.000000","reverse_dns_suffix":"google.com (rate-limited-proxy-***-***-***-***.google.com)","verification_method":"Reverse DNS to rate-limited-proxy-***-***-***-***.google.com with matching forward DNS, or match the source IP against special-crawlers.json.","docs_url":"https://developers.google.com/crawling/docs/crawlers-fetchers/overview-google-crawlers","page_url":"https://www.proxies.sx/crawlers/adsbot-mobile-web"}
{"slug":"google-safety","name":"Google-Safety","operator":"Google","user_agent_token":"Google-Safety","purpose_category":"other","robots_token":"undocumented","robots_token_is_policy_only":false,"honors_robots_txt":"no","ip_ranges_url":"https://developers.google.com/static/crawling/ipranges/special-crawlers.json","ip_prefix_count":272,"ip_ranges_creation_time":"2026-09-04T14:45:59.000000","reverse_dns_suffix":"google.com (rate-limited-proxy-***-***-***-***.google.com)","verification_method":"Reverse DNS to rate-limited-proxy-***-***-***-***.google.com with matching forward DNS, or match the source IP against special-crawlers.json.","docs_url":"https://developers.google.com/crawling/docs/crawlers-fetchers/overview-google-crawlers","page_url":"https://www.proxies.sx/crawlers/google-safety"}
{"slug":"googleother","name":"GoogleOther","operator":"Google","user_agent_token":"GoogleOther","purpose_category":"other","robots_token":"GoogleOther","robots_token_is_policy_only":false,"honors_robots_txt":"yes","ip_ranges_url":"https://developers.google.com/static/crawling/ipranges/common-crawlers.json","ip_prefix_count":317,"ip_ranges_creation_time":"2026-09-04T14:46:55.000000","reverse_dns_suffix":"googlebot.com (crawl-***-***-***-***.googlebot.com or geo-crawl-***-***-***-***.geo.googlebot.com)","verification_method":"Reverse DNS to googlebot.com, google.com, or googleusercontent.com with matching forward DNS, or match the source IP against the published common-crawlers.json ranges.","docs_url":"https://developers.google.com/crawling/docs/crawlers-fetchers/overview-google-crawlers","page_url":"https://www.proxies.sx/crawlers/googleother"}
{"slug":"googleother-image","name":"GoogleOther-Image","operator":"Google","user_agent_token":"GoogleOther-Image","purpose_category":"other","robots_token":"GoogleOther-Image","robots_token_is_policy_only":false,"honors_robots_txt":"yes","ip_ranges_url":"https://developers.google.com/static/crawling/ipranges/common-crawlers.json","ip_prefix_count":317,"ip_ranges_creation_time":"2026-09-04T14:46:55.000000","reverse_dns_suffix":"googlebot.com (crawl-***-***-***-***.googlebot.com or geo-crawl-***-***-***-***.geo.googlebot.com)","verification_method":"Reverse DNS to googlebot.com, google.com, or googleusercontent.com with matching forward DNS, or match the source IP against the published common-crawlers.json ranges.","docs_url":"https://developers.google.com/crawling/docs/crawlers-fetchers/overview-google-crawlers","page_url":"https://www.proxies.sx/crawlers/googleother-image"}
{"slug":"googleother-video","name":"GoogleOther-Video","operator":"Google","user_agent_token":"GoogleOther-Video","purpose_category":"other","robots_token":"GoogleOther-Video","robots_token_is_policy_only":false,"honors_robots_txt":"yes","ip_ranges_url":"https://developers.google.com/static/crawling/ipranges/common-crawlers.json","ip_prefix_count":317,"ip_ranges_creation_time":"2026-09-04T14:46:55.000000","reverse_dns_suffix":"googlebot.com (crawl-***-***-***-***.googlebot.com or geo-crawl-***-***-***-***.geo.googlebot.com)","verification_method":"Reverse DNS to googlebot.com, google.com, or googleusercontent.com with matching forward DNS, or match the source IP against the published common-crawlers.json ranges.","docs_url":"https://developers.google.com/crawling/docs/crawlers-fetchers/overview-google-crawlers","page_url":"https://www.proxies.sx/crawlers/googleother-video"}
{"slug":"meta-externalads","name":"Meta-ExternalAds","operator":"Meta","user_agent_token":"meta-externalads","purpose_category":"other","robots_token":"meta-externalads","robots_token_is_policy_only":false,"honors_robots_txt":"yes","ip_ranges_url":"","ip_prefix_count":"","ip_ranges_creation_time":"","reverse_dns_suffix":"","verification_method":"Meta does not document a verification method for this crawler. The web crawlers page gives only a page-wide statement to allow-list \"the user agent strings or the IP addresses (more secure)\", with no IP file, ASN, or reverse-DNS suffix given.","docs_url":"https://developers.facebook.com/docs/sharing/webmasters/web-crawlers","page_url":"https://www.proxies.sx/crawlers/meta-externalads"}
{"slug":"adidxbot","name":"AdIdxBot","operator":"Microsoft Bing","user_agent_token":"adidxbot/2.0","purpose_category":"other","robots_token":"AdIdxbot","robots_token_is_policy_only":false,"honors_robots_txt":"undocumented","ip_ranges_url":"","ip_prefix_count":"","ip_ranges_creation_time":"","reverse_dns_suffix":"","verification_method":"Microsoft does not document a verification method specific to AdIdxBot. The crawlers article states for all Bing crawlers: \"To ensure the IP is authentic, use the Verify Bingbot tool or learn about all verification methods we offer\", those methods being reverse DNS to search.msn.com and the published list at https://www.bing.com/toolbox/bingbot.json. Whether AdIdxBot IPs are included in bingbot.json or resolve under search.msn.com is not stated.","docs_url":"https://www.bing.com/webmasters/help/which-crawlers-does-bing-use-8c184ec0","page_url":"https://www.proxies.sx/crawlers/adidxbot"}
{"slug":"oai-adsbot","name":"OAI-AdsBot","operator":"OpenAI","user_agent_token":"OAI-AdsBot","purpose_category":"other","robots_token":"undocumented","robots_token_is_policy_only":false,"honors_robots_txt":"undocumented","ip_ranges_url":"https://openai.com/adsbot.json","ip_prefix_count":2,"ip_ranges_creation_time":"2026-05-12T03:00:00.000000","reverse_dns_suffix":"","verification_method":"OpenAI publishes IP ranges for OAI-AdsBot at https://openai.com/adsbot.json. No reverse-DNS method is documented.","docs_url":"https://developers.openai.com/api/docs/bots","page_url":"https://www.proxies.sx/crawlers/oai-adsbot"}
{"slug":"yadirectfetcher","name":"YaDirectFetcher","operator":"Yandex","user_agent_token":"YaDirectFetcher","purpose_category":"other","robots_token":"YaDirectFetcher","robots_token_is_policy_only":false,"honors_robots_txt":"no","ip_ranges_url":"","ip_prefix_count":"","ip_ranges_creation_time":"","reverse_dns_suffix":"yandex.ru, yandex.net, yandex.com","verification_method":"Reverse DNS lookup on the requesting IP must return a host ending in yandex.ru, yandex.net or yandex.com, and a forward DNS lookup of that host must return the same IP. Yandex Webmaster also provides an IP address check tool.","docs_url":"https://yandex.com/support/webmaster/en/robot-workings/check-yandex-robots.html","page_url":"https://www.proxies.sx/crawlers/yadirectfetcher"}
{"slug":"yandex","name":"Yandex","operator":"Yandex","user_agent_token":"Yandex","purpose_category":"other","robots_token":"Yandex","robots_token_is_policy_only":true,"honors_robots_txt":"partial","ip_ranges_url":"","ip_prefix_count":"","ip_ranges_creation_time":"","reverse_dns_suffix":"","verification_method":"The operator does not document a verification method.","docs_url":"https://yandex.com/support/webmaster/en/robot-workings/check-yandex-robots.html","page_url":"https://www.proxies.sx/crawlers/yandex"}
{"slug":"yandexaccessibilitybot","name":"YandexAccessibilityBot","operator":"Yandex","user_agent_token":"YandexAccessibilityBot","purpose_category":"other","robots_token":"YandexAccessibilityBot","robots_token_is_policy_only":false,"honors_robots_txt":"partial","ip_ranges_url":"","ip_prefix_count":"","ip_ranges_creation_time":"","reverse_dns_suffix":"yandex.ru, yandex.net, yandex.com","verification_method":"Reverse DNS lookup on the requesting IP must return a host ending in yandex.ru, yandex.net or yandex.com, and a forward DNS lookup of that host must return the same IP. Yandex Webmaster also provides an IP address check tool.","docs_url":"https://yandex.com/support/webmaster/en/robot-workings/check-yandex-robots.html","page_url":"https://www.proxies.sx/crawlers/yandexaccessibilitybot"}
{"slug":"yandexadnet","name":"YandexAdNet","operator":"Yandex","user_agent_token":"YandexAdNet","purpose_category":"other","robots_token":"YandexAdNet","robots_token_is_policy_only":false,"honors_robots_txt":"yes","ip_ranges_url":"","ip_prefix_count":"","ip_ranges_creation_time":"","reverse_dns_suffix":"yandex.ru, yandex.net, yandex.com","verification_method":"Reverse DNS lookup on the requesting IP must return a host ending in yandex.ru, yandex.net or yandex.com, and a forward DNS lookup of that host must return the same IP. Yandex Webmaster also provides an IP address check tool.","docs_url":"https://yandex.com/support/webmaster/en/robot-workings/check-yandex-robots.html","page_url":"https://www.proxies.sx/crawlers/yandexadnet"}
{"slug":"yandexcheckbot","name":"YandexCheckBot","operator":"Yandex","user_agent_token":"YandexCheckBot","purpose_category":"other","robots_token":"YandexCheckBot","robots_token_is_policy_only":false,"honors_robots_txt":"partial","ip_ranges_url":"","ip_prefix_count":"","ip_ranges_creation_time":"","reverse_dns_suffix":"yandex.ru, yandex.net, yandex.com","verification_method":"Reverse DNS lookup on the requesting IP must return a host ending in yandex.ru, yandex.net or yandex.com, and a forward DNS lookup of that host must return the same IP. Yandex Webmaster also provides an IP address check tool.","docs_url":"https://yandex.com/support/webmaster/en/robot-workings/check-yandex-robots.html","page_url":"https://www.proxies.sx/crawlers/yandexcheckbot"}
{"slug":"yandexdirect","name":"YandexDirect","operator":"Yandex","user_agent_token":"YandexDirect","purpose_category":"other","robots_token":"YandexDirect","robots_token_is_policy_only":false,"honors_robots_txt":"partial","ip_ranges_url":"","ip_prefix_count":"","ip_ranges_creation_time":"","reverse_dns_suffix":"yandex.ru, yandex.net, yandex.com","verification_method":"Reverse DNS lookup on the requesting IP must return a host ending in yandex.ru, yandex.net or yandex.com, and a forward DNS lookup of that host must return the same IP. Yandex Webmaster also provides an IP address check tool.","docs_url":"https://yandex.com/support/webmaster/en/robot-workings/check-yandex-robots.html","page_url":"https://www.proxies.sx/crawlers/yandexdirect"}
{"slug":"yandexdirectdyn","name":"YandexDirectDyn","operator":"Yandex","user_agent_token":"YandexDirectDyn","purpose_category":"other","robots_token":"YandexDirectDyn","robots_token_is_policy_only":false,"honors_robots_txt":"partial","ip_ranges_url":"","ip_prefix_count":"","ip_ranges_creation_time":"","reverse_dns_suffix":"yandex.ru, yandex.net, yandex.com","verification_method":"Reverse DNS lookup on the requesting IP must return a host ending in yandex.ru, yandex.net or yandex.com, and a forward DNS lookup of that host must return the same IP. Yandex Webmaster also provides an IP address check tool.","docs_url":"https://yandex.com/support/webmaster/en/robot-workings/check-yandex-robots.html","page_url":"https://www.proxies.sx/crawlers/yandexdirectdyn"}
{"slug":"yandeximageresizer","name":"YandexImageResizer","operator":"Yandex","user_agent_token":"YandexImageResizer","purpose_category":"other","robots_token":"YandexImageResizer","robots_token_is_policy_only":false,"honors_robots_txt":"yes","ip_ranges_url":"","ip_prefix_count":"","ip_ranges_creation_time":"","reverse_dns_suffix":"yandex.ru, yandex.net, yandex.com","verification_method":"Reverse DNS lookup on the requesting IP must return a host ending in yandex.ru, yandex.net or yandex.com, and a forward DNS lookup of that host must return the same IP. Yandex Webmaster also provides an IP address check tool.","docs_url":"https://yandex.com/support/webmaster/en/robot-workings/check-yandex-robots.html","page_url":"https://www.proxies.sx/crawlers/yandeximageresizer"}
{"slug":"yandexmarket","name":"YandexMarket","operator":"Yandex","user_agent_token":"YandexMarket","purpose_category":"other","robots_token":"YandexMarket","robots_token_is_policy_only":false,"honors_robots_txt":"partial","ip_ranges_url":"","ip_prefix_count":"","ip_ranges_creation_time":"","reverse_dns_suffix":"yandex.ru, yandex.net, yandex.com","verification_method":"Reverse DNS lookup on the requesting IP must return a host ending in yandex.ru, yandex.net or yandex.com, and a forward DNS lookup of that host must return the same IP. Yandex Webmaster also provides an IP address check tool.","docs_url":"https://yandex.com/support/webmaster/en/robot-workings/check-yandex-robots.html","page_url":"https://www.proxies.sx/crawlers/yandexmarket"}
{"slug":"yandexmetrika","name":"YandexMetrika","operator":"Yandex","user_agent_token":"YandexMetrika","purpose_category":"other","robots_token":"YandexMetrika","robots_token_is_policy_only":false,"honors_robots_txt":"no","ip_ranges_url":"","ip_prefix_count":"","ip_ranges_creation_time":"","reverse_dns_suffix":"yandex.ru, yandex.net, yandex.com","verification_method":"Reverse DNS lookup on the requesting IP must return a host ending in yandex.ru, yandex.net or yandex.com, and a forward DNS lookup of that host must return the same IP. Yandex Webmaster also provides an IP address check tool.","docs_url":"https://yandex.com/support/webmaster/en/robot-workings/check-yandex-robots.html","page_url":"https://www.proxies.sx/crawlers/yandexmetrika"}
{"slug":"yandexmobilescreenshotbot","name":"YandexMobileScreenShotBot","operator":"Yandex","user_agent_token":"YandexMobileScreenShotBot","purpose_category":"other","robots_token":"YandexMobileScreenShotBot","robots_token_is_policy_only":false,"honors_robots_txt":"partial","ip_ranges_url":"","ip_prefix_count":"","ip_ranges_creation_time":"","reverse_dns_suffix":"yandex.ru, yandex.net, yandex.com","verification_method":"Reverse DNS lookup on the requesting IP must return a host ending in yandex.ru, yandex.net or yandex.com, and a forward DNS lookup of that host must return the same IP. Yandex Webmaster also provides an IP address check tool.","docs_url":"https://yandex.com/support/webmaster/en/robot-workings/check-yandex-robots.html","page_url":"https://www.proxies.sx/crawlers/yandexmobilescreenshotbot"}
{"slug":"yandexontodbapi","name":"YandexOntoDBAPI","operator":"Yandex","user_agent_token":"YandexOntoDBAPI","purpose_category":"other","robots_token":"YandexOntoDBAPI","robots_token_is_policy_only":false,"honors_robots_txt":"partial","ip_ranges_url":"","ip_prefix_count":"","ip_ranges_creation_time":"","reverse_dns_suffix":"yandex.ru, yandex.net, yandex.com","verification_method":"Reverse DNS lookup on the requesting IP must return a host ending in yandex.ru, yandex.net or yandex.com, and a forward DNS lookup of that host must return the same IP. Yandex Webmaster also provides an IP address check tool.","docs_url":"https://yandex.com/support/webmaster/en/robot-workings/check-yandex-robots.html","page_url":"https://www.proxies.sx/crawlers/yandexontodbapi"}
{"slug":"yandexpartner","name":"YandexPartner","operator":"Yandex","user_agent_token":"YandexPartner","purpose_category":"other","robots_token":"YandexPartner","robots_token_is_policy_only":false,"honors_robots_txt":"partial","ip_ranges_url":"","ip_prefix_count":"","ip_ranges_creation_time":"","reverse_dns_suffix":"yandex.ru, yandex.net, yandex.com","verification_method":"Reverse DNS lookup on the requesting IP must return a host ending in yandex.ru, yandex.net or yandex.com, and a forward DNS lookup of that host must return the same IP. Yandex Webmaster also provides an IP address check tool.","docs_url":"https://yandex.com/support/webmaster/en/robot-workings/check-yandex-robots.html","page_url":"https://www.proxies.sx/crawlers/yandexpartner"}
{"slug":"yandexrca","name":"YandexRCA","operator":"Yandex","user_agent_token":"YandexRCA","purpose_category":"other","robots_token":"YandexRCA","robots_token_is_policy_only":false,"honors_robots_txt":"partial","ip_ranges_url":"","ip_prefix_count":"","ip_ranges_creation_time":"","reverse_dns_suffix":"yandex.ru, yandex.net, yandex.com","verification_method":"Reverse DNS lookup on the requesting IP must return a host ending in yandex.ru, yandex.net or yandex.com, and a forward DNS lookup of that host must return the same IP. Yandex Webmaster also provides an IP address check tool.","docs_url":"https://yandex.com/support/webmaster/en/robot-workings/check-yandex-robots.html","page_url":"https://www.proxies.sx/crawlers/yandexrca"}
{"slug":"yandexscreenshotbot","name":"YandexScreenshotBot","operator":"Yandex","user_agent_token":"YandexScreenshotBot","purpose_category":"other","robots_token":"YandexScreenshotBot","robots_token_is_policy_only":false,"honors_robots_txt":"partial","ip_ranges_url":"","ip_prefix_count":"","ip_ranges_creation_time":"","reverse_dns_suffix":"yandex.ru, yandex.net, yandex.com","verification_method":"Reverse DNS lookup on the requesting IP must return a host ending in yandex.ru, yandex.net or yandex.com, and a forward DNS lookup of that host must return the same IP. Yandex Webmaster also provides an IP address check tool.","docs_url":"https://yandex.com/support/webmaster/en/robot-workings/check-yandex-robots.html","page_url":"https://www.proxies.sx/crawlers/yandexscreenshotbot"}
{"slug":"yandexspravbot","name":"YandexSpravBot","operator":"Yandex","user_agent_token":"YandexSpravBot","purpose_category":"other","robots_token":"YandexSpravBot","robots_token_is_policy_only":false,"honors_robots_txt":"yes","ip_ranges_url":"","ip_prefix_count":"","ip_ranges_creation_time":"","reverse_dns_suffix":"yandex.ru, yandex.net, yandex.com","verification_method":"Reverse DNS lookup on the requesting IP must return a host ending in yandex.ru, yandex.net or yandex.com, and a forward DNS lookup of that host must return the same IP. Yandex Webmaster also provides an IP address check tool.","docs_url":"https://yandex.com/support/webmaster/en/robot-workings/check-yandex-robots.html","page_url":"https://www.proxies.sx/crawlers/yandexspravbot"}
{"slug":"yandextracker","name":"YandexTracker","operator":"Yandex","user_agent_token":"YandexTracker","purpose_category":"other","robots_token":"YandexTracker","robots_token_is_policy_only":false,"honors_robots_txt":"partial","ip_ranges_url":"","ip_prefix_count":"","ip_ranges_creation_time":"","reverse_dns_suffix":"yandex.ru, yandex.net, yandex.com","verification_method":"Reverse DNS lookup on the requesting IP must return a host ending in yandex.ru, yandex.net or yandex.com, and a forward DNS lookup of that host must return the same IP. Yandex Webmaster also provides an IP address check tool.","docs_url":"https://yandex.com/support/webmaster/en/robot-workings/check-yandex-robots.html","page_url":"https://www.proxies.sx/crawlers/yandextracker"}