Googlebot
Google operates Googlebot, the generic name for the Googlebot Smartphone and Googlebot Desktop crawlers behind Google Search, Discover, Images, Video and News. It obeys robots.txt under the Googlebot token. Site owners verify it by reverse DNS to googlebot.com, google.com or googleusercontent.com with a matching forward lookup, or against the published common-crawlers.json IP ranges.
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)What Googlebot does
Googlebot Smartphone shows up in logs with an Android user agent string such as Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html), while Googlebot Desktop uses a Chrome desktop string with the same Googlebot/2.1 token. Two shorter forms appear rarely. The Chrome version in the string is a placeholder that increases over time, so match it with wildcards. Google warns that this user agent header is often spoofed, which is why log filtering alone does not prove a request came from Google.
Google states that common crawlers always obey robots.txt rules when crawling automatically. Rules addressed to the Googlebot token affect Google Search, including Discover and every Search feature, plus Google Images, Google Video and Google News. Blocking the token affects all of those products at once, and because Smartphone and Desktop share the same token you cannot allow one while refusing the other. For Search, the crawler reads the first 2MB of a supported file type and the first 64MB of a PDF; each referenced CSS or JavaScript resource is fetched separately under the same limit.
Googlebot supports HTTP/1.1 and HTTP/2, accepts gzip, deflate and br encoding, and uses HTTP caching through ETag with If-None-Match and Last-Modified with If-Modified-Since. When it crawls from US IPs its timezone is Pacific Time. The IP range file at developers.google.com/static/crawling/ipranges/common-crawlers.json lists creationTime and prefixes as ipv4Prefix or ipv6Prefix entries. The former googlebot.json path under /static/search/apis/ipranges/ is no longer linked from the docs, although Google's changelog says it will keep working for the time being.
Operator note. Listed as a common crawler: 'They always obey robots.txt rules when crawling automatically.' Other documented UA strings are the desktop 'Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/W.X.Y.Z Safari/537.36' and, rarely, 'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)' and 'Googlebot/2.1 (+http://www.google.com/bot.html)', where Chrome/W.X.Y.Z is a placeholder for the Chrome version that increases over time, so use wildcards when matching. Smartphone and desktop Googlebot obey the same robots.txt token and cannot be targeted separately; for Google Search it crawls the first 2MB of a supported file type and the first 64MB of a PDF, with each referenced resource (CSS, JavaScript) fetched separately under the same limit; it supports HTTP/1.1 and HTTP/2, gzip/deflate/br, and HTTP caching via ETag/If-None-Match and Last-Modified/If-Modified-Since; its timezone is Pacific Time when crawling from US IPs; and the docs warn that the user-agent header is often spoofed. The IP range list moved to /crawling/ipranges on 2026-02-11, and the old https://developers.google.com/static/search/apis/ipranges/googlebot.json redirects to /static/crawling/ipranges/common-crawlers.json, which the changelog says 'will continue to work for the time being'.
Controlling Googlebot with robots.txt
Use the token Googlebot in robots.txt. Google documents that Googlebot honors robots.txt directives.
User-agent: Googlebot
Disallow: /User-agent: Googlebot
Allow: /Verifying a request is really Googlebot
Anyone can put Googlebot in a User-Agent header. A reverse DNS lookup of the source IP must resolve to googlebot.com, google.com, or googleusercontent.com, and a forward DNS lookup of that name must return the same IP. Alternatively, match the IP against the published common-crawlers.json CIDR ranges.
host <request-ip>
# expect a hostname ending in googlebot.com (crawl-***-***-***-***.googlebot.com or geo-crawl-***-***-***-***.geo.googlebot.com)
host <that-hostname>
# expect it to resolve back to <request-ip>Google publishes a machine-readable list. Our probe read it on 2026-09-06; the file's own creation time was 2026-09-04.
- Prefixes
- 317
- IPv4
- 170
- IPv6
- 147
First entries at fetch time: 2001:4860:4801:10::/64, 2001:4860:4801:11::/64, 2001:4860:4801:12::/64
SHA-256 of the fetched body: 241e712f9c902b33aa4c0cc79268c34cb00c025edbc27298547be298763ddade
Common questions
Should I block Googlebot?
Only if you accept the effect on Google Search, Discover, Google Images, Google Video and Google News, because rules addressed to the Googlebot token affect all of them. Smartphone and Desktop share the token, so a block applies to both. Rules under the Googlebot token in robots.txt are the documented control.
How do I verify Googlebot?
Run a reverse DNS lookup on the source IP; the hostname must end in googlebot.com, google.com or googleusercontent.com, typically crawl-***-***-***-***.googlebot.com or geo-crawl-***-***-***-***.geo.googlebot.com. Then run a forward lookup on that hostname and confirm it returns the same IP. As an alternative, match the IP against the CIDR prefixes in Google's published common-crawlers.json.
Does Googlebot run JavaScript?
Google's documentation says Googlebot fetches each referenced resource, including JavaScript files, separately under the 2MB limit that applies to supported file types. Google's documentation does not state whether the fetched scripts are executed, so treat JavaScript execution as undocumented here.
Sources
Every fact on this page was checked against Google's own documentation, listed below, and re-checked by a second reviewer before publication. Reviewed September 2026.
- https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers
- https://developers.google.com/search/docs/crawling-indexing/googlebot
- https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests
- https://developers.google.com/crawling/docs/crawlers-fetchers/overview-google-crawlers
- https://developers.google.com/static/crawling/ipranges/common-crawlers.json
- https://developers.google.com/static/search/apis/ipranges/googlebot.json
- https://developers.google.com/crawling/docs/changelog
Related crawlers
This registry documents how operators describe their own bots so site owners can identify and control them. It does not publish third-party IP lists or guess at undocumented behaviour. To see how your own site responds to automated visitors, the bot detection scanner reads a URL's live response and names the protection it finds.