Verify Googlebot and Bingbot by IP Address

Start with a trustworthy source IP

Behind a CDN or reverse proxy, the application's immediate peer can be that intermediary. Determine how your edge records the original client address and which forwarding headers it overwrites. An arbitrary request header supplied by the caller is not a reliable identity source.

Mozilla's X-Forwarded-For guidance explains why security decisions require a trusted proxy configuration. Establish that boundary before performing a correct lookup on a potentially incorrect address.

Keep the user-agent as a claimed identity in the log, alongside the independently observed address. Give the verification result its own field so reports can distinguish a claimed bot from a verified network match. The crawler registry provides operator references and dated snapshots for this research step.

Choose the operator verification method

Google's verification guide distinguishes its crawler and fetcher categories and publishes corresponding ranges. Bing's verification guidance describes the DNS method and an IP-list alternative, with daily refresh guidance for that list.

For a one-off investigation, manual DNS checks are easy to inspect. For repeated classification, use a validated range cache or a cached DNS verification service with explicit timeouts and result age. Choose based on your operational needs; a synchronous external DNS request on every incoming web request can become an availability dependency.

Check reverse and forward DNS together

Reverse DNS asks which name is associated with an address. The forward check asks whether that name resolves back to the same address. Preserve both answers and compare normalized IP values, including IPv6, rather than relying on how the strings happen to be formatted.

# Replace SOURCE_IP with the trusted address from your request log.
host SOURCE_IP

# Check the returned name against the operator's documented suffix.
# Then resolve that exact returned name:
host RETURNED_HOSTNAME

# Confirm an A or AAAA answer equals the original SOURCE_IP.

Validate the suffix at a DNS-label boundary. A hostname containing googlebot.com somewhere in its text is insufficient. For example, googlebot.com.attacker.example does not end with the Googlebot domain. Normalize case and remove the final DNS dot before checking.

def has_dns_suffix(hostname, suffix):
    name = hostname.rstrip(".").lower()
    expected = suffix.rstrip(".").lower()
    return name == expected or name.endswith("." + expected)

assert has_dns_suffix("crawl.googlebot.com.", "googlebot.com")
assert not has_dns_suffix("googlebot.com.attacker.example", "googlebot.com")
assert not has_dns_suffix("fakegooglebot.com", "googlebot.com")

This helper performs only the suffix check. It does not query DNS or verify a crawler by itself. Use the permitted suffix for the selected operator category, then require forward confirmation. A timeout should produce an unknown result with a reason, not an invented positive answer.

Use the correct crawler range list

Fetch the list linked by the operator's current documentation and retain its retrieval time and hash. Google's common crawler reference is a useful starting point for Googlebot. Avoid replacing a crawler-specific publication with all Google Cloud or Azure address space.

Parse IPv4 and IPv6 entries, validate the complete file, and match the observed address against the CIDRs. The Python CIDR guide explains the membership operation and why parsers must follow the selected publisher's schema. Its downloadable AWS parser cannot be used unchanged for a Googlebot or Bingbot document.

Retain the last accepted version if an update fails, but mark its age. Decide whether a stale match is sufficient for the particular action. A recently validated file can support a different decision from a cache that has not refreshed for weeks.

Our Googlebot record and Bingbot record link to their sources. They are reference pages, not a live verification service for an IP submitted by a visitor.

Separate verification from access policy

Store at least the claimed crawler, source IP, verification method, result, check time and source version. Keep a reason for unknown or failed results. This gives a reviewer enough context to distinguish a stale list, a malformed address and a genuine non-match.

Then decide what the verified identity is allowed to access. A match should not automatically bypass account authorization or make private URLs public. Test any crawler rule against public and authenticated routes before deployment.

Also inspect the content the crawler receives. An allowed connection that gets a challenge page or an error body can still fail your publishing objective. Verification answers who the request appears to come from; it does not establish that a search engine indexed the page. Use the research library to trace the underlying network records and operator references.

Common questions

Can I trust the user-agent alone?

Treat it as a claim. Use the operator's verification procedure and a source address obtained through your trusted edge configuration.

Does robots.txt verify a crawler?

No. Crawling instructions and source verification perform different jobs. Keep the identity check separate from the rules for fetching public content.

Does a failed lookup prove impersonation?

Not necessarily. A timeout, stale cache or wrong source IP can leave the result unknown. Preserve the failure reason before assigning a malicious-traffic label.