CCBot
Common Crawl, a non-profit foundation, operates CCBot to build an open repository of web crawl data that anyone can access and analyze. CCBot honors robots.txt under the token CCBot and obeys Crawl-delay. Site owners verify it against the published IP range file at index.commoncrawl.org/ccbot.json and, for IPv4, forward-confirmed reverse DNS to a host under crawl.commoncrawl.org.
CCBot/2.0 (https://commoncrawl.org/faq/)What CCBot does
CCBot identifies itself as CCBot/2.0 (https://commoncrawl.org/faq/); an older form was CCBot/1.0 (+https://commoncrawl.org/bot.html), and Common Crawl says it may increment the version in future. The crawler is Nutch-based and runs on Apache Hadoop, using Map-Reduce to pick crawl candidates from its crawl database. Common Crawl describes its dataset as a sample of the web; it does not generally archive an entire website, only a randomly selected subset of it. The bot fetches over HTTP/1.1 and HTTP/2 (HTTP/2 only over TLS), on IPv4 and IPv6, supports conditional GET and gzip, Brotli and ZStandard compression, and does not execute JavaScript or use cookies.
Common Crawl says CCBot checks robots.txt first and fetches a page with HTTP GET only if crawling is allowed. Blocking it with User-agent: CCBot and Disallow: / keeps the site out of the open crawl archive; Common Crawl periodically rechecks robots.txt for updates. The bot obeys Crawl-delay, honors the nofollow link attribute, reads Sitemap lines in robots.txt, follows up to four consecutive redirects (five for robots.txt), backs off adaptively on HTTP 429 and 5xx responses, and by default waits a few seconds between requests to the same site. Common Crawl also has an opt-out registry, the Opt-Out Ledger, in addition to robots.txt.
The IP range file is a JSON object with synctoken, creationTime, notes and a prefixes array whose entries carry either ipv4Prefix or ipv6Prefix. As of 2026-08-11 it held five prefixes, one IPv6 block and four IPv4 blocks, and the FAQ lists the same CIDRs inline. Common Crawl runs CCBot on dedicated ranges with reverse DNS on IPv4, for example 18.97.14.84 resolving to 18-97-14-84.crawl.commoncrawl.org and back to the same address, but reverse DNS is not yet supported over IPv6. The operator warns that other crawlers falsely identify themselves as CCBot and recommends verifying the User-Agent string.
Operator note. "CCBot is an automated crawler, checking first the robots.txt, and if crawling a page is allowed, fetches pages using HTTP GET requests." It can be blocked with "User-agent: CCBot / Disallow: /", and "We will periodically continue to check if the robots.txt file has been updated." It honors Crawl-delay ("We obey the Crawl-delay parameter for robots.txt", for example User-agent: CCBot / Crawl-delay: 2), the nofollow link attribute and Sitemap directives in robots.txt, and an opt-out registry (Opt-Out Ledger) is offered in addition to robots.txt. Fetching uses HTTP/1.1 and HTTP/2 (HTTP/2 only over TLS) over IPv4 and IPv6, follows up to four consecutive redirects (five for robots.txt per RFC 9309), supports conditional GET and gzip, Brotli and ZStandard compression, backs off adaptively on HTTP 429 and 5xx responses, by default waits a few seconds between requests to the same site, and "Currently, JavaScript is not executed and Cookies are not used." An older user-agent string is "CCBot/1.0 (+https://commoncrawl.org/bot.html)", and "We may increment the version number in the future."
Controlling CCBot with robots.txt
Use the token CCBot in robots.txt. Common Crawl documents that CCBot honors robots.txt directives.
User-agent: CCBot
Disallow: /User-agent: CCBot
Allow: /Verifying a request is really CCBot
Anyone can put CCBot in a User-Agent header. Common Crawl publishes its IP ranges at https://index.commoncrawl.org/ccbot.json and lists the same CIDRs inline in its FAQ, where the IPv6 range is labelled "IPv6 (no reverse DNS)"; the file's notes read: "IP ranges used by CCBot. For verification of IPv4 addresses, FCrDNS is also recommended." The operator states: "CCBot is now run on dedicated IP address ranges with reverse DNS (except over IPv6 where reverse DNS is not yet supported.)" and gives the example that 18.97.14.84 resolves to 18-97-14-84.crawl.commoncrawl.org, which resolves back to 18.97.14.84. The documentation warns: "we are aware of crawlers falsely identifying themselves as CCBot. We recommend verifying UserAgent strings to ensure authenticity."
host <request-ip>
# expect a hostname ending in .crawl.commoncrawl.org
host <that-hostname>
# expect it to resolve back to <request-ip>Common Crawl publishes a machine-readable list. Our probe read it on 2026-09-06; the file's own creation time was 2026-08-11.
- Prefixes
- 5
- IPv4
- 4
- IPv6
- 1
First entries at fetch time: 2600:1f28:365:8000::/56, 3.41.188.32/29, 18.97.9.168/29
SHA-256 of the fetched body: 0a4c3b6707326d7f083139a979062604a57edfbb251bd29f23f5a86c5107ea30
Common questions
Should I block CCBot?
Blocking it removes your site from the Common Crawl open dataset, which the foundation maintains so anyone can access and analyze web crawl data. Common Crawl says CCBot obeys robots.txt and Crawl-delay, so User-agent: CCBot with Disallow: / stops it and Crawl-delay: 2 slows it. Common Crawl also has an Opt-Out Ledger beyond robots.txt.
Does CCBot run JavaScript?
No. Common Crawl states that JavaScript is not executed and cookies are not used. The crawler fetches pages with plain HTTP GET requests over HTTP/1.1 or HTTP/2, supports conditional GET and compressed responses, and follows up to four consecutive redirects, or five for robots.txt.
How do I verify CCBot?
Check the requesting IP against the range file at index.commoncrawl.org/ccbot.json, which lists both IPv4 and IPv6 prefixes. For IPv4, also run a reverse DNS lookup: a real CCBot host resolves to a name under crawl.commoncrawl.org that resolves back to the same IP. Reverse DNS is not available over IPv6. Common Crawl warns that other crawlers falsely claim the CCBot name.
Sources
Every fact on this page was checked against Common Crawl's own documentation, listed below, and re-checked by a second reviewer before publication. Reviewed September 2026.
This registry documents how operators describe their own bots so site owners can identify and control them. It does not publish third-party IP lists or guess at undocumented behaviour. To see how your own site responds to automated visitors, the bot detection scanner reads a URL's live response and names the protection it finds.