How to Stop ByteSpider (TikTok/ByteDance) from Scraping Server Bandwidth
Prevent ByteSpider (TikTok/ByteDance LLM crawler) from aggressive crawling, spiking server CPU, and consuming bandwidth using robots.txt and WAF rules.
Displayed below main page header or above the tool container. • Zero CLS Container
Automate & Test This in Our Free Tool
Eliminate syntax errors and test live URLs client-side using our dedicated LLMs.txt & AI Crawler Directive Generator.
The Technical Problem & Root Cause
ByteDance's LLM bot (ByteSpider) aggressively crawls dynamic endpoints without respecting crawl-delay, spiking CPU utilization and proxy transfer fees.
High CPU usage alert / Vercel Edge Requests spike from User-Agent: Mozilla/5.0... ByteSpider
Production-Grade Solution & Code Snippet
Copy and paste this verified configuration directly into your project:
# robots.txt User-agent: ByteSpider Disallow: / User-agent: CCBot Disallow: / User-agent: GPTBot Disallow: / # Allow AI Search & Attribution Engines User-agent: OAI-SearchBot Allow: / User-agent: PerplexityBot Allow: / User-agent: * Allow: / Sitemap: https://omniseotools.com/sitemap.xml
Step-by-Step Implementation Walkthrough
11. Append Explicit ByteSpider Disallow in public/robots.txt
Append explicit User-agent: ByteSpider Disallow block to public/robots.txt.
22. Add Cloudflare WAF Custom Expression
Add Cloudflare WAF custom expression: (http.user_agent contains "ByteSpider") -> Block.
33. Verify Using Crawler Verification Tools
Verify using client-side crawler verification tools and server access logs.
Separates the interactive tool output from the deep technical guide. • Zero CLS Container
- Believing blocking ByteSpider affects regular organic search indexing on Google or Bing.
- Relying solely on robots.txt for scrapers that deliberately spoof standard browser user agents.
- Blocking TikTok's official sharing preview bot (Bytespider vs TikTokBot) if you rely on TikTok in-app link preview cards.
- Placing crawl-delay in robots.txt for bots that completely ignore crawl-delay specifications.
Frequently Asked Questions
Does ByteSpider respect standard robots.txt Disallow directives?▼
Official ByteDance ByteSpider crawler nodes generally respect robots.txt Disallow instructions within 24-48 hours. However, third-party scrapers mimicking ByteSpider user agents may ignore robots.txt, making edge WAF blocking at Cloudflare or AWS WAF the most reliable solution.
How do I block ByteSpider at the Nginx or Apache server level?▼
In Nginx: if ($http_user_agent ~* 'ByteSpider') { return 403; }. In Apache (.htaccess): RewriteEngine On RewriteCond %{HTTP_USER_AGENT} ByteSpider [NC] RewriteRule .* - [F,L].
Does blocking ByteSpider affect my TikTok Ads or TikTok marketing tracking tags?▼
No. ByteSpider is an automated web crawler used for ByteDance LLMs and search indexing. TikTok Pixel tracking (events.js) and TikTok in-app ad conversions are executed client-side on user devices and are entirely unaffected by robots.txt rules.
Related Tools & Next Workflow Steps
Complementary utilities to streamline your SEO audit, indexing, and content strategy.
Robots.txt Generator & Validator
Generate, test, and validate standard-compliant robots.txt files with live syntax checking, multi-user-agent rules, and sitemap directives.
XML Sitemap Generator & Validator
Generate standard XML sitemaps from URL batches, customize crawl priorities, and validate existing sitemap XML client-side.
Schema Markup Generator
Generate Google-compliant JSON-LD Schema.org structured data markup for Articles, Products, Organizations, and WebSites.
Resource Hint & Preconnect Generator
Generate and validate preload, preconnect, dns-prefetch, and prefetch tags for Next.js, HTML, and HTTP headers to optimize Core Web Vitals.
Displayed below main page header or above the tool container. • Zero CLS Container