Feedfetcher
Google operates Feedfetcher, a user-triggered fetcher that crawls RSS or Atom feeds for Google News and WebSub. Because a user requests the fetch, it generally ignores robots.txt, and Google documents no robots token for it. Site owners verify it by reverse DNS with a matching forward lookup, or by checking the IP against Google's published user-triggered fetcher JSON files.
FeedFetcher-Google; (+http://www.google.com/feedfetcher.html)What Feedfetcher does
Requests from Feedfetcher carry the user-agent "FeedFetcher-Google; (+http://www.google.com/feedfetcher.html)". Google places it in the user-triggered fetcher category, which covers requests made because a user asked for a resource. The documented purpose is crawling RSS or Atom feeds for Google News and for WebSub. Google does not document how often it fetches a feed, how many feeds one visit covers, or whether it runs JavaScript, so server logs are the only source for those details on a given site.
Google's documentation says user-triggered fetchers "generally ignore robots.txt rules" because the fetch was requested by a user, and it lists no robots.txt token for Feedfetcher. A Disallow line will not reliably stop these requests. A site owner who wants to stop feed fetches has to act at the server level, for example by returning an error for the feed URL to verified Feedfetcher addresses. Blocking removes the feed from Google News ingestion and WebSub delivery; allowing it keeps those feeds current.
Google publishes two IP range files for this category. user-triggered-fetchers.json holds user-owned addresses that resolve to gae.googleusercontent.com; user-triggered-fetchers-google.json holds Google-controlled addresses that resolve to google.com in the form google-proxy-***-***-***-***.google.com. The documentation does not state which of the two files Feedfetcher specifically uses, so a verification check should consult both files before deciding whether a request is genuine.
Operator note. User-triggered fetcher: 'Because the fetch was requested by a user, these fetchers generally ignore robots.txt rules.' The docs do not state which of the two fetcher JSON files Feedfetcher specifically uses; the category as a whole publishes both.
Controlling Feedfetcher with robots.txt
Use the token undocumented in robots.txt. Google documents that this fetcher does not consult robots.txt, so a robots rule will not stop it; see the operator note.
User-agent: undocumented
Disallow: /User-agent: undocumented
Allow: /Verifying a request is really Feedfetcher
Anyone can put FeedFetcher-Google in a User-Agent header. Reverse DNS with matching forward DNS, or match the source IP against user-triggered-fetchers.json (user-owned, resolves to gae.googleusercontent.com) or user-triggered-fetchers-google.json (Google-controlled, resolves to google.com).
host <request-ip>
# expect a hostname ending in gae.googleusercontent.com (***-***-***-***.gae.googleusercontent.com) or google.com (google-proxy-***-***-***-***.google.com)
host <that-hostname>
# expect it to resolve back to <request-ip>Google publishes a machine-readable list. Our probe read it on 2026-09-06; the file's own creation time was 2026-09-04.
- Prefixes
- 496
- IPv4
- 248
- IPv6
- 248
First entries at fetch time: 2001:4860:4801:4004::/64, 2001:4860:4801:4008::/64, 2001:4860:4801:4009::/64
SHA-256 of the fetched body: 5590a1a5d2fbe83872191aa2d8d45e9c854c56c49b742df9d33b7612f5482a5a
Common questions
Does Feedfetcher obey robots.txt?
No, in general. Google says user-triggered fetchers generally ignore robots.txt rules because a user requested the fetch, and it documents no robots token for Feedfetcher. If you need to stop it, respond at the server level for the feed URL instead of relying on a Disallow rule.
How do I verify Feedfetcher?
Run a reverse DNS lookup on the requesting IP and confirm the hostname ends in gae.googleusercontent.com or google.com, then run a forward lookup and check that it returns the same IP. Alternatively, match the IP against the prefixes in user-triggered-fetchers.json or user-triggered-fetchers-google.json; Google does not say which file Feedfetcher uses, so check both.
Does Feedfetcher run JavaScript?
The operator's documentation does not say. Google describes Feedfetcher only as a fetcher of RSS or Atom feeds for Google News and WebSub, and its published notes say nothing about script execution for this fetcher. Read your own logs to see what it requests.
Sources
Every fact on this page was checked against Google's own documentation, listed below, and re-checked by a second reviewer before publication. Reviewed September 2026.
- https://developers.google.com/crawling/docs/crawlers-fetchers/google-user-triggered-fetchers
- https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests
- https://developers.google.com/static/crawling/ipranges/user-triggered-fetchers-google.json
- https://developers.google.com/static/crawling/ipranges/user-triggered-fetchers.json
Related crawlers
This registry documents how operators describe their own bots so site owners can identify and control them. It does not publish third-party IP lists or guess at undocumented behaviour. To see how your own site responds to automated visitors, the bot detection scanner reads a URL's live response and names the protection it finds.