Bytespider
ByteDance operates Bytespider, which its Chinese webmaster documentation describes as the crawler of Toutiao Search, following links to crawl pages and build a search index. The documentation never mentions robots.txt for Bytespider, so no robots token or compliance statement is documented. Site owners verify it by reverse DNS, where genuine hostnames end in .bytedance.com; the same page lists ten IPv4 ranges.
Mozilla/5.0 (compatible; Bytespider; https://zhanzhang.toutiao.com/) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.0.0 Safari/537.36What Bytespider does
Bytespider arrives with one of three documented user-agent strings, for PC Chrome, Android and iPhone, each containing the token Bytespider with a capital B and a link to zhanzhang.toutiao.com. ByteDance describes a classic search pipeline behind it. The crawler follows hyperlinks to take page snapshots, then processes each page by extracting keywords, de-duplicating, segmenting Chinese text, classifying the page type and analysing links to compute importance, and Toutiao Search then shows the title, URL and snippet in results. The documentation states nothing further about how the data is used.
Allowing Bytespider lets a site appear in Toutiao Search results, which is the only use the operator states. Blocking it removes the site from that index. The About Bytespider page never mentions robots.txt and names no User-agent token. The same webmaster platform has a separate document on robots matching semantics, covering the * and $ wildcards and path rules that follow Google's conventions, but that document does not name Bytespider or promise compliance. Site owners should therefore treat robots.txt handling as undocumented for this crawler.
Site owners adjust crawl rate through the platform's crawl-frequency tool. ByteDance says Bytespider adjusts its frequency to the site's content update rate and server load, that settings apply per site with no parent-site concept, and that changes take effect within one day, or at the earliest the next day. For verification, the documentation gives a worked reverse-DNS example resolving to bytespider-111-225-148-148.crawl.bytedance.com and says any hostname outside the *.bytedance.com pattern is an impostor. The page lists ten IPv4 /24 ranges inline, but ByteDance publishes no machine-readable IP file. The documentation does not mention JavaScript execution. Spider feedback goes to zhanzhang@bytedance.com.
Operator note. ByteDance documents three official user-agent variants - desktop, Android ('Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; Bytespider; https://zhanzhang.toutiao.com/)') and iOS ('Mozilla/5.0 (iPhone; CPU iPhone OS 7_1_2 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko) Version/7.0 Mobile Safari/537.36 (compatible; Bytespider; https://zhanzhang.toutiao.com/)') - and stresses that the name is written with a capital B (首写字母为大写). The 'About Bytespider' page does not mention robots.txt or state a robots.txt User-agent token; the webmaster platform's separate 'Robots匹配说明' (Robots matching explanation) document (https://zhanzhang.toutiao.com/docs/intro/520) states that its robots matching supports the * and $ wildcards, that path matching follows Google's rules (only rules starting with / or * can match), and that URL-parameter matching is unordered with wildcards allowed in keys and values, but it does not name Bytespider or state a compliance commitment. Crawl rate is controlled through the platform's '抓取频次' (crawl frequency) tool (https://zhanzhang.toutiao.com/docs/intro/26892): Bytespider adjusts its frequency automatically based on the site's content update rate and server load, the About page says webmaster settings take effect within one day (我们会在1天内时间内生效), the crawl-frequency document says they take effect at the earliest the next day (最快次日生效), and settings are per site with no parent-site concept (无主站概念). The documentation is Chinese-only, the spider feedback contact is zhanzhang@bytedance.com, the ten IP ranges are listed inline on the page rather than in a downloadable file (the document says '总共涉及10个ip'), and nothing is said about JavaScript execution.
Controlling Bytespider with robots.txt
Use the token undocumented in robots.txt. ByteDance's documentation does not state whether this bot honors robots.txt.
User-agent: undocumented
Disallow: /User-agent: undocumented
Allow: /Verifying a request is really Bytespider
Anyone can put Bytespider in a User-Agent header. ByteDance's official documentation says webmasters can verify Bytespider by reverse DNS lookup (host, nslookup -qt=ptr, or dig -x): genuine Bytespider hostnames follow the *.bytedance.com pattern, and anything else is an impostor (冒充). The documentation gives the worked example host 111.225.148.250 -> bytespider-111-225-148-148.crawl.bytedance.com. The same page lists ten IPv4 /24 ranges inline (110.249.201.0/24, 110.249.202.0/24, 111.225.148.0/24, 111.225.149.0/24, 220.243.135.0/24, 220.243.136.0/24, 220.243.188.0/24, 220.243.189.0/24, 60.8.123.0/24, 60.8.151.0/24); no machine-readable JSON or TXT IP range file is published.
host <request-ip>
# expect a hostname ending in .bytedance.com
host <that-hostname>
# expect it to resolve back to <request-ip>ByteDance lists addresses on its documentation page rather than in a machine-readable file. Our probe read that page on 2026-09-06 and extracted the entries below.
- Prefixes
- 12
- IPv4
- 12
- IPv6
- 0
First entries at fetch time: 110.249.201.0, 110.249.202.0, 111.225.148.0
SHA-256 of the fetched body: f9997cde8d850d486b7d7bc2b5b31460c26e8352c47df0cdb252753d58329f3e
Common questions
Should I block Bytespider?
Blocking it removes your site from Toutiao Search, which is the only purpose ByteDance documents. Because the official page never mentions robots.txt for Bytespider, a robots.txt rule has no documented effect. If you want to exclude it, block at the server level using the .bytedance.com reverse-DNS check or the ten IPv4 ranges ByteDance lists.
How do I verify Bytespider?
Run a reverse DNS lookup on the requesting IP with host, nslookup or dig -x. ByteDance says genuine hostnames match *.bytedance.com, for example bytespider-111-225-148-148.crawl.bytedance.com, and anything else is an impostor. You can also compare the IP against the ten /24 ranges listed on the About Bytespider page, though no downloadable IP file exists.
Does Bytespider respect robots.txt?
The operator's documentation does not say. The About Bytespider page never mentions robots.txt or a User-agent token. A separate robots-matching document on the same platform explains wildcard and path semantics but does not name Bytespider or commit it to compliance, so treat robots.txt handling as undocumented.
Sources
Every fact on this page was checked against ByteDance's own documentation, listed below, and re-checked by a second reviewer before publication. Reviewed September 2026.
Related crawlers
This registry documents how operators describe their own bots so site owners can identify and control them. It does not publish third-party IP lists or guess at undocumented behaviour. To see how your own site responds to automated visitors, the bot detection scanner reads a URL's live response and names the protection it finds.