Meta-ExternalAgent

In access logs the crawler identifies itself as meta-externalagent/1.1, either on its own or followed by a documentation path in parentheses. Meta's web crawlers page states that it fetches pages for use cases such as training foundation AI models, or improving Meta products by indexing content directly. Meta does not document how often pages are refetched, the crawl rate, or whether the crawler executes JavaScript, so a site's own log data is the only picture of how it behaves on a given host.

Because Meta lists the purpose as AI training, allowing the crawler means a site's content can be used to train Meta's foundation models and to feed product indexing. Blocking it withdraws the site from those uses, as far as Meta's stated policy goes. Meta's own robots.txt example uses this exact token, allowing everything and then disallowing a /private/ directory, which shows that Meta expects path-level rules rather than only a blanket allow or deny. Meta's page says that by configuring robots.txt a site owner can specify how Meta's web crawlers should interact with the site.

Meta caches robots.txt for up to 24 hours, so a new rule can take a full day to take effect. On verification, the crawler page only says that site owners can allow-list either the user-agent strings or, as the more secure option, the IP addresses. It gives no IP file, ASN, or reverse-DNS suffix for Meta-ExternalAgent, so there is no documented way to confirm that a request carrying this user agent came from Meta.