Meta-WebIndexer

In access logs the crawler identifies itself as meta-webindexer/1.1 followed, in parentheses, by a path to Meta's web crawlers documentation, and Meta also documents the shorter form meta-webindexer/1.1 on its own. Meta describes the visits as navigating the web so that Meta AI search result quality improves for its users. Meta does not document how often the crawler returns to a site or whether it executes JavaScript, so the documentation gives a site owner no basis for assuming that rendered content is read or that it is skipped.

Allowing the crawler keeps the site's pages available to Meta AI search. Blocking it removes the site from the results that Meta AI search draws on, per Meta's description of the crawler's purpose. Meta points site owners to robots.txt as the control and says to add a Disallow rule for the relevant crawler name. Meta states that it relies on industry-standard robots.txt rather than non-standard formats such as NoAI tags. The page's only worked example uses the meta-externalagent token, so the meta-webindexer token follows the same instruction rather than appearing in a literal robots.txt sample.

Meta documents no verification method for this crawler. Its page tells site owners to allow-list either the user-agent strings or the IP addresses, calling the IP route more secure, but the crawler page gives no IP file, ASN or reverse-DNS suffix for Meta-WebIndexer. Meta's AS32934 whois command and geofeed CSV appear only in webhook documentation, not on the crawler page, so the user-agent string is the only identifier Meta documents for this crawler.