User-triggered fetcher · Google
Chrome Web Store user agent, IP ranges and robots.txt
Chrome Web Store is Google's user-triggered fetcher. It identifies itself with the Google-CWS user-agent token, does not consult robots.txt according to its operator, and runs from 496 published IP prefixes.
- Operator
- Reviewed
- September 2026
- Sources
- 1
- Operator
- User-agent token
- Google-CWS
- Purpose
- User-triggered fetcher
- robots.txt token
- undocumented
- Honors robots.txt
- No, per the operator
- Published IP ranges
- Yes, 496 prefixes (248 IPv4, 248 IPv6)
- Verification
- Reverse DNS with matching forward DNS, or match the source IP against user-triggered-fetchers.json or user-triggered-fetchers-google.json.
- Reverse DNS suffix
- gae.googleusercontent.com (***-***-***-***.gae.googleusercontent.com) or google.com (google-proxy-***-***-***-***.google.com)
- Full user-agent example
- Mozilla/5.0 (compatible; Google-CWS)
Block or allow Chrome Web Store in robots.txt
User-agent: undocumented
Disallow: /User-agent: undocumented
Allow: /Verify a request is really Chrome Web Store
Anyone can put Google-CWS in a User-Agent header. Reverse DNS with matching forward DNS, or match the source IP against user-triggered-fetchers.json or user-triggered-fetchers-google.json.
host <request-ip>
# expect a hostname ending in gae.googleusercontent.com (***-***-***-***.gae.googleusercontent.com) or google.com (google-proxy-***-***-***-***.google.com)
host <that-hostname>
# expect it to resolve back to <request-ip>Published IP ranges
Google publishes a machine-readable list. Our probe read it on 2026-09-06; the file's own creation time was 2026-09-04.
- Prefixes
- 496
- IPv4
- 248
- IPv6
- 248
First entries at fetch time: 2001:4860:4801:4004::/64, 2001:4860:4801:4008::/64, 2001:4860:4801:4009::/64
SHA-256 of the fetched body: 5590a1a5d2fbe83872191aa2d8d45e9c854c56c49b742df9d33b7612f5482a5a
What Chrome Web Store does
Google's Chrome Web Store fetcher, Google-CWS, requests URLs that developers provide in the metadata of their Chrome extensions and themes. As a user-triggered fetcher it generally ignores robots.txt, and Google documents no robots token for it. Verify it by reverse DNS to gae.googleusercontent.com or google.com with matching forward DNS, or against Google's published IP range file.
In access logs the fetcher appears with the user agent Mozilla/5.0 (compatible; Google-CWS). Google describes it as a user-triggered fetcher, meaning a request happens because an extension or theme developer entered a URL in the metadata of their Chrome extension or theme, and the store fetches that URL. The visits are tied to individual developer actions rather than to a scheduled crawl of the site. Google does not document request frequency or volume for this fetcher.
Because Google classes it as user-triggered, the fetcher generally ignores robots.txt, and Google lists no robots.txt token for it. Blocking it has to happen at the server or firewall level using the user agent string or the IP ranges. Doing so stops the Chrome Web Store from retrieving URLs that developers pointed at your site. Google does not document what the store does with the fetched content, so its documentation does not say what a block changes for those developers' listings. Allowing it lets the store fetch the URLs those developers supplied.
Verification uses reverse DNS. The host name should end in gae.googleusercontent.com, in the form ***-***-***-***.gae.googleusercontent.com, or in google.com as google-proxy-***-***-***-***.google.com, and a forward lookup of that host name should return the original IP address. Google also publishes the fetcher's IP addresses in user-triggered-fetchers.json and user-triggered-fetchers-google.json, so a firewall or log filter can match against those lists instead of running DNS lookups for every request. Treat a request that carries the Google-CWS user agent but fails both checks as unverified.
Operator note. User-triggered fetcher; generally ignores robots.txt.
Common questions
- Does Google-CWS obey robots.txt?
- Generally no. Google groups it with user-triggered fetchers, which it says generally ignore robots.txt, and the documentation lists no robots.txt token for it. To control it, filter on the Google-CWS user agent or on Google's published IP ranges at the server level.
- How do I verify a Google-CWS request?
- Run a reverse DNS lookup on the requesting IP. A real request resolves to a host ending in gae.googleusercontent.com or google.com, and a forward lookup of that host returns the same IP. Alternatively, compare the IP with Google's user-triggered-fetchers.json or user-triggered-fetchers-google.json files.
- Why is the Chrome Web Store fetching pages from my site?
- Google says the fetcher requests URLs that developers provide in the metadata of their Chrome extensions and themes. Someone listing an extension or theme in the Chrome Web Store has pointed a metadata field at a URL on your site, and the store retrieves it. Google does not document what it does with the fetched content.
Sources
The operator documentation used for this entry. Recorded review date September 2026; consult the linked pages for current policy and compare them with this dated snapshot.
This registry documents how operators describe their own bots so site owners can identify and control them. It does not publish third-party IP lists or guess at undocumented behaviour. To see how your own site responds to automated visitors, the bot detection scanner reads a URL's live response and names the protection it finds.