Amazonbot user agent, IP ranges and robots.txt

Amazon operates Amazonbot, which its documentation says is used to improve Amazon products and services, to give customers more accurate information, and may be used to train Amazon AI models. It honors robots.txt under the token Amazonbot. Site owners verify it against Amazon's published IP address list; Amazon does not document a reverse DNS suffix.

In access logs the crawler identifies itself with the token Amazonbot inside a browser-like user agent string, for example "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amazonbot/0.1) Chrome/W.X.Y.Z Safari/537.36". Amazon describes the visits as a way to improve its products and services, which it says helps it provide more accurate information to customers, and the fetched content may be used to train Amazon AI models. This registry classes it as AI training for that reason. Amazon does not document whether the crawler executes JavaScript.

Allowing Amazonbot lets Amazon fetch your pages for those uses, including possible model training. Blocking it with a Disallow rule for the Amazonbot token stops those fetches. Amazon also says that sites which allow Amazonbot in robots.txt may be eligible for the Amazon Content Partners program at contentpartners.amazon.com. If you want a page fetched but not used for model training, Amazon says it respects a page-level meta robots noarchive tag, which it reads as "do not use the page for model training". It also honors noindex and none at page level and rel=nofollow at link level.

Amazon's documentation spells out its robots.txt handling. The crawler fetches robots.txt per host, so site.example.com/robots.txt is read separately from example.com/robots.txt, and it may use a cached copy from the last 30 days. If robots.txt cannot be fetched, the crawler behaves as if the file does not exist. It does not support the crawl-delay directive, and Amazon says settings changes can take around 24 hours to take effect. Questions go to amazonbot@amazon.com.

Operator note. Amazon's documentation states that Amazonbot respects the Robots Exclusion Protocol, honoring user-agent and allow/disallow directives; it fetches the host-level robots.txt or uses a cached copy from the last 30 days, checks robots.txt per host (for example site.example.com/robots.txt separately), and behaves as if robots.txt does not exist when it cannot be fetched. It respects link-level rel=nofollow and the page-level meta robots directives noarchive ('do not use the page for model training'), noindex and none, does not support the crawl-delay directive, and settings changes may take about 24 hours to take effect. The contact address is amazonbot@amazon.com, and the documentation says sites allowing Amazonbot in robots.txt may be eligible for the Amazon Content Partners program (contentpartners.amazon.com). JavaScript execution is not documented.