Google StoreBot
Google operates Storebot-Google, a common crawler whose crawling preferences affect every part of Google Shopping, including the Shopping tab in Google Search and Google Shopping itself. It honors the Storebot-Google robots.txt token. Site owners verify requests by reverse DNS to googlebot.com, google.com or googleusercontent.com with a matching forward lookup, or against Google's published common-crawlers.json ranges.
Mozilla/5.0 (X11; Linux x86_64; Storebot-Google/1.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Safari/537.36What Google StoreBot does
In logs the desktop agent reads Mozilla/5.0 (X11; Linux x86_64; Storebot-Google/1.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Safari/537.36, and the mobile agent reads Mozilla/5.0 (Linux; Android 8.0; Pixel 2 Build/OPD3.170816.012; Storebot-Google/1.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36. The Chrome version portion is a placeholder in Google's documentation. Both strings carry the Storebot-Google/1.0 token, which is the part to match on. Google lists it among its common crawlers.
Google says crawling preferences addressed to Storebot-Google affect every part of Google Shopping, including the Shopping tab in Google Search and Google Shopping. Allowing the crawler lets Google fetch the pages it uses for those Shopping products. A Disallow rule for the Storebot-Google user agent in robots.txt withdraws that access and, by Google's description, applies across all of Google Shopping rather than to a single product. Google honors the token, so robots.txt is the documented control. Google's overview also notes that Storebot-Google only supports caching in certain conditions, without listing what those conditions are.
To confirm a request is from Google, run a reverse DNS lookup on the source IP and check that the hostname ends in googlebot.com, google.com or googleusercontent.com; the common-crawler form is crawl-***-***-***-***.googlebot.com or geo-crawl-***-***-***-***.geo.googlebot.com. Then resolve that hostname forward and confirm it returns the same IP. Google also publishes the IP ranges for its common crawlers in common-crawlers.json, so matching the source address against that file is an equivalent check.
Operator note. Common crawler. Mobile agent: 'Mozilla/5.0 (Linux; Android 8.0; Pixel 2 Build/OPD3.170816.012; Storebot-Google/1.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36'. The overview notes 'Storebot-Google only supports caching in certain conditions.'
Controlling Google StoreBot with robots.txt
Use the token Storebot-Google in robots.txt. Google documents that Google StoreBot honors robots.txt directives.
User-agent: Storebot-Google
Disallow: /User-agent: Storebot-Google
Allow: /Verifying a request is really Google StoreBot
Anyone can put Storebot-Google in a User-Agent header. Reverse DNS to googlebot.com, google.com, or googleusercontent.com with matching forward DNS, or match the source IP against the published common-crawlers.json ranges.
host <request-ip>
# expect a hostname ending in googlebot.com (crawl-***-***-***-***.googlebot.com or geo-crawl-***-***-***-***.geo.googlebot.com)
host <that-hostname>
# expect it to resolve back to <request-ip>Google publishes a machine-readable list. Our probe read it on 2026-09-06; the file's own creation time was 2026-09-04.
- Prefixes
- 317
- IPv4
- 170
- IPv6
- 147
First entries at fetch time: 2001:4860:4801:10::/64, 2001:4860:4801:11::/64, 2001:4860:4801:12::/64
SHA-256 of the fetched body: 241e712f9c902b33aa4c0cc79268c34cb00c025edbc27298547be298763ddade
Common questions
Should I block Storebot-Google?
Only if you want your pages out of Google Shopping. Google says crawling preferences for this user agent affect every part of Google Shopping, including the Shopping tab in Google Search, so a Disallow rule removes the pages from those products. Google's documentation describes no narrower effect; the decision covers Google Shopping as a whole.
How do I verify Storebot-Google?
Reverse-resolve the source IP; a genuine request maps to a hostname ending in googlebot.com, google.com or googleusercontent.com, in the form crawl-***-***-***-***.googlebot.com. Forward-resolve that hostname and confirm it returns the same IP. Alternatively, compare the IP with Google's published common-crawlers.json ranges.
Does Storebot-Google cache content?
Google's crawler overview says Storebot-Google only supports caching in certain conditions and does not list what those conditions are. Google's documentation for this crawler gives no further detail on caching, so that one sentence is the full extent of what Google has published on the subject.
Sources
Every fact on this page was checked against Google's own documentation, listed below, and re-checked by a second reviewer before publication. Reviewed September 2026.
- https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers
- https://developers.google.com/crawling/docs/crawlers-fetchers/overview-google-crawlers
- https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests
- https://developers.google.com/static/crawling/ipranges/common-crawlers.json
Related crawlers
This registry documents how operators describe their own bots so site owners can identify and control them. It does not publish third-party IP lists or guess at undocumented behaviour. To see how your own site responds to automated visitors, the bot detection scanner reads a URL's live response and names the protection it finds.