Meta-ExternalFetcher
Meta-ExternalFetcher is Meta's fetcher for individual links requested by a user; Meta says it supports product functions such as evaluating and improving agentic AI capabilities, including helping AI navigate websites to complete tasks. Its robots.txt token is meta-externalfetcher, but Meta states it may bypass robots.txt for user-requested fetches. Meta documents no IP list or reverse DNS suffix for verifying it.
meta-externalfetcher/1.1 (+/documentation/sharing/webmasters/web-crawlers)What Meta-ExternalFetcher does
Requests from this fetcher show a User-Agent beginning with meta-externalfetcher, and Meta documents the form meta-externalfetcher/1.1 followed by a path to its web crawlers documentation. Meta says the fetcher retrieves individual links at a user's request, so Meta ties each fetch to something a person did in a Meta product. Meta describes the product side as evaluating and improving agentic AI capabilities, which includes helping AI navigate websites to complete tasks for users, so a hit on a page means a Meta product asked for that specific page on a user's behalf.
Meta's page tells site owners to control a crawler by adding a disallow for the relevant crawler name, so the token to use is meta-externalfetcher. Meta then adds that this crawler may bypass robots.txt because it performs fetches that were requested by the user. A disallow rule is therefore a stated preference rather than a guarantee. Blocking it at the server level removes the page from whatever a Meta user was trying to do with it, such as an AI completing a task on the site; allowing it keeps those user-driven fetches working.
Verification is thin. Meta's crawler page makes one general statement, that owners can allow-list the user agent strings or the IP addresses, calling the IP route more secure, but it publishes no IP file, ASN or reverse DNS suffix for this fetcher. Whether it executes JavaScript is also undocumented. Site owners can only match on the user agent string, which is the less secure of the two options by Meta's own description, and should treat that match as an identification hint rather than proof of origin.
Operator note. A second documented user-agent form is "meta-externalfetcher/1.1". The documentation states: "Accordingly, this crawler may bypass robots.txt rules." and "The Meta-ExternalFetcher crawler may bypass robots.txt because it performs fetches that were requested by the user." It is used for user-initiated fetches, and the robots.txt token is the crawler name per the page's instruction to "add a disallow for the relevant crawler". JavaScript execution is undocumented.
Controlling Meta-ExternalFetcher with robots.txt
Use the token meta-externalfetcher in robots.txt. Meta documents limits on how this bot applies robots.txt; see the operator note.
User-agent: meta-externalfetcher
Disallow: /User-agent: meta-externalfetcher
Allow: /Verifying a request is really Meta-ExternalFetcher
Anyone can put meta-externalfetcher in a User-Agent header. Meta does not document a verification method for this crawler. The web crawlers page gives only a page-wide statement to allow-list "the user agent strings or the IP addresses (more secure)", with no IP file, ASN, or reverse-DNS suffix given.
Common questions
Should I block Meta-ExternalFetcher?
That depends on whether you want Meta's AI features to open your pages when a user asks. Meta says the fetcher retrieves individual links at a user's request and helps AI navigate websites to complete tasks, so blocking it stops those user-driven visits. A robots.txt disallow for meta-externalfetcher records your preference, but Meta states the fetcher may bypass robots.txt for user-requested fetches.
Does Meta-ExternalFetcher respect robots.txt?
Only partly. Meta's documentation says the crawler may bypass robots.txt rules because it performs fetches that were requested by the user. You can still add a disallow for meta-externalfetcher, and Meta's page describes that as the way to address a specific crawler, but Meta does not promise the fetcher will honor it on every request.
How do I verify Meta-ExternalFetcher?
Meta does not document a verification method specific to this fetcher. Its crawler page says site owners can allow-list the user agent strings or the IP addresses and calls the IP option more secure, but it publishes no IP file, ASN or reverse DNS suffix. The only documented identifier is the user agent string, which begins with meta-externalfetcher/1.1.
Sources
Every fact on this page was checked against Meta's own documentation, listed below, and re-checked by a second reviewer before publication. Reviewed September 2026.
Related crawlers
This registry documents how operators describe their own bots so site owners can identify and control them. It does not publish third-party IP lists or guess at undocumented behaviour. To see how your own site responds to automated visitors, the bot detection scanner reads a URL's live response and names the protection it finds.