Chrome Web Store
Google's Chrome Web Store fetcher, Google-CWS, requests URLs that developers provide in the metadata of their Chrome extensions and themes. As a user-triggered fetcher it generally ignores robots.txt, and Google documents no robots token for it. Verify it by reverse DNS to gae.googleusercontent.com or google.com with matching forward DNS, or against Google's published IP range file.
Mozilla/5.0 (compatible; Google-CWS)What Chrome Web Store does
In access logs the fetcher appears with the user agent Mozilla/5.0 (compatible; Google-CWS). Google describes it as a user-triggered fetcher, meaning a request happens because an extension or theme developer entered a URL in the metadata of their Chrome extension or theme, and the store fetches that URL. The visits are tied to individual developer actions rather than to a scheduled crawl of the site. Google does not document request frequency or volume for this fetcher.
Because Google classes it as user-triggered, the fetcher generally ignores robots.txt, and Google lists no robots.txt token for it. Blocking it has to happen at the server or firewall level using the user agent string or the IP ranges. Doing so stops the Chrome Web Store from retrieving URLs that developers pointed at your site. Google does not document what the store does with the fetched content, so its documentation does not say what a block changes for those developers' listings. Allowing it lets the store fetch the URLs those developers supplied.
Verification uses reverse DNS. The host name should end in gae.googleusercontent.com, in the form ***-***-***-***.gae.googleusercontent.com, or in google.com as google-proxy-***-***-***-***.google.com, and a forward lookup of that host name should return the original IP address. Google also publishes the fetcher's IP addresses in user-triggered-fetchers.json and user-triggered-fetchers-google.json, so a firewall or log filter can match against those lists instead of running DNS lookups for every request. Treat a request that carries the Google-CWS user agent but fails both checks as unverified.
Operator note. User-triggered fetcher; generally ignores robots.txt.
Controlling Chrome Web Store with robots.txt
Use the token undocumented in robots.txt. Google documents that this fetcher does not consult robots.txt, so a robots rule will not stop it; see the operator note.
User-agent: undocumented
Disallow: /User-agent: undocumented
Allow: /Verifying a request is really Chrome Web Store
Anyone can put Google-CWS in a User-Agent header. Reverse DNS with matching forward DNS, or match the source IP against user-triggered-fetchers.json or user-triggered-fetchers-google.json.
host <request-ip>
# expect a hostname ending in gae.googleusercontent.com (***-***-***-***.gae.googleusercontent.com) or google.com (google-proxy-***-***-***-***.google.com)
host <that-hostname>
# expect it to resolve back to <request-ip>Google publishes a machine-readable list. Our probe read it on 2026-09-06; the file's own creation time was 2026-09-04.
- Prefixes
- 496
- IPv4
- 248
- IPv6
- 248
First entries at fetch time: 2001:4860:4801:4004::/64, 2001:4860:4801:4008::/64, 2001:4860:4801:4009::/64
SHA-256 of the fetched body: 5590a1a5d2fbe83872191aa2d8d45e9c854c56c49b742df9d33b7612f5482a5a
Common questions
Does Google-CWS obey robots.txt?
Generally no. Google groups it with user-triggered fetchers, which it says generally ignore robots.txt, and the documentation lists no robots.txt token for it. To control it, filter on the Google-CWS user agent or on Google's published IP ranges at the server level.
How do I verify a Google-CWS request?
Run a reverse DNS lookup on the requesting IP. A real request resolves to a host ending in gae.googleusercontent.com or google.com, and a forward lookup of that host returns the same IP. Alternatively, compare the IP with Google's user-triggered-fetchers.json or user-triggered-fetchers-google.json files.
Why is the Chrome Web Store fetching pages from my site?
Google says the fetcher requests URLs that developers provide in the metadata of their Chrome extensions and themes. Someone listing an extension or theme in the Chrome Web Store has pointed a metadata field at a URL on your site, and the store retrieves it. Google does not document what it does with the fetched content.
Sources
Every fact on this page was checked against Google's own documentation, listed below, and re-checked by a second reviewer before publication. Reviewed September 2026.
Related crawlers
This registry documents how operators describe their own bots so site owners can identify and control them. It does not publish third-party IP lists or guess at undocumented behaviour. To see how your own site responds to automated visitors, the bot detection scanner reads a URL's live response and names the protection it finds.