FacebookExternalHit
Meta operates FacebookExternalHit, which crawls the content of an app or website shared on Facebook, Instagram or Messenger. The robots.txt token is facebookexternalhit, and Meta says the crawler may bypass robots.txt for security or integrity checks. Meta documents no reverse-DNS check and publishes no IP list; it advises allowlisting the user agent or the crawler's IP addresses.
facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)What FacebookExternalHit does
The crawler identifies itself as "facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)" or the shorter "facebookexternalhit/1.1". Meta's stated purpose is to crawl the content of an app or website that someone shared on Facebook, Instagram or Messenger, so a request follows a share on one of those apps. Meta's requirements page says the server must support gzip and deflate encodings and that Open Graph properties must appear within the first 1 MB of the response. The site should either honor the Range header by returning all required properties within the requested bytes or ignore the Range header entirely. Meta's troubleshooting section reproduces the request with curl using a Range header of bytes=0-524288.
Allowing the crawler lets Meta read the shared page and the Open Graph properties it requires. Blocking it stops that fetch for links shared on Meta's apps. Meta says a disallow for the relevant crawler in robots.txt is the way to block it, with the user agent product token as the name, and that changes can take up to 24 hours to apply because crawlers may cache robots.txt for that long. Meta also warns that FacebookExternalHit might bypass robots.txt when performing security or integrity checks, such as checking for malware or malicious content, so a disallow is not absolute.
Verification is thin. The current crawler page only says to allowlist the user agent strings or, more securely, the IP addresses the crawler uses, without listing them. An archived 2024 version of the retired Facebook Crawler page gave a whois command against AS32934 on whois.radb.net and warned that the addresses change often. Meta currently publishes a variant of that command on its WhatsApp webhooks page, describing the result as Meta's webhook servers rather than crawlers, and links a geofeed CSV at facebook.com/peering/geofeed. Meta documents no reverse-DNS hostname and does not say whether the crawler executes JavaScript. The contact address is webmasters@meta.com.
Operator note. A second documented user-agent form is "facebookexternalhit/1.1", and the documentation states: "Note that the FacebookExternalHit crawler might bypass robots.txt when performing security or integrity checks, such as checking for malware or malicious content." The current page requires that the server support gzip and deflate encodings, that Open Graph properties appear within the first 1 MB of the response, and that "Your app or website should either generate and return a response with all required properties according to the bytes specified in the Range header of the crawler request or it should ignore the Range header altogether."; its Troubleshooting section gives a simulated request: curl -v --compressed -H "Range: bytes=0-524288" -H "Connection: close" -A "facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)" "$URL". On robots.txt the page states: "Please allow up to 24 hours for changes to robots.txt to take effect because crawlers may cache the contents of robots.txt for up to 24 hours.", the robots.txt token is the user-agent product token per the page's instruction "In order to block these crawlers, add a disallow for the relevant crawler to robots.txt." (the page's only worked example uses meta-externalagent), and the contact is webmasters@meta.com. JavaScript execution is undocumented, and the retired Facebook Crawler page additionally listed a user agent "facebookcatalog/1.0" that is absent from the current page.
Controlling FacebookExternalHit with robots.txt
Use the token facebookexternalhit in robots.txt. Meta documents limits on how this bot applies robots.txt; see the operator note.
User-agent: facebookexternalhit
Disallow: /User-agent: facebookexternalhit
Allow: /Verifying a request is really FacebookExternalHit
Anyone can put facebookexternalhit in a User-Agent header. Meta's current crawler page gives no method beyond "Add to your allow list either the user agent strings or the IP addresses (more secure) used by the crawler." The retired Facebook Crawler page documented: "To get a current list of IP addresses the crawler uses, run the following command. whois -h whois.radb.net -- '-i origin AS32934' | grep ^route" and "These IP addresses change often." Meta currently publishes a variant of that command on its WhatsApp webhooks page, described as returning "the IP addresses of Meta's webhook servers" rather than crawlers: "whois -h whois.radb.net -- '-i origin AS32934' | grep '^route' | awk '{print $2}' | sort", and links a geofeed CSV at https://facebook.com/peering/geofeed ("download a CSV that lists Meta's IP addresses"); no reverse-DNS method is documented.
Common questions
Should I block facebookexternalhit?
Blocking it stops Meta's apps from fetching your pages when users share them. Meta says to add a disallow for the crawler in robots.txt and allow up to 24 hours for the change to take effect. The crawler may still bypass robots.txt for security or integrity checks, so the block is not absolute.
How do I verify facebookexternalhit?
Meta's current page gives no verification method beyond allowlisting the user agent or IP addresses. The retired Facebook Crawler page documented a whois query for routes announced by AS32934, noting the addresses change often, and Meta links a geofeed CSV at facebook.com/peering/geofeed. Meta documents no reverse-DNS hostname for this crawler.
Why does facebookexternalhit send a Range header?
Meta's troubleshooting example sends Range: bytes=0-524288 with a Connection: close header, and its requirements refer to the Range header of the crawler request. Meta says your site should either return all required Open Graph properties within the requested byte range or ignore the Range header altogether. Open Graph properties must also appear within the first 1 MB of the response.
Sources
Every fact on this page was checked against Meta's own documentation, listed below, and re-checked by a second reviewer before publication. Reviewed September 2026.
- https://developers.facebook.com/docs/sharing/webmasters/web-crawlers
- https://developers.facebook.com/documentation/sharing/webmasters/web-crawlers
- https://developers.facebook.com/documentation/business-messaging/whatsapp/webhooks/overview
- https://web.archive.org/web/20240101000413id_/https://developers.facebook.com/docs/sharing/webmasters/crawler/
Related crawlers
This registry documents how operators describe their own bots so site owners can identify and control them. It does not publish third-party IP lists or guess at undocumented behaviour. To see how your own site responds to automated visitors, the bot detection scanner reads a URL's live response and names the protection it finds.