Rate-Limiting Bytespider in Nginx

Some crawlers do not obey robots.txt. ByteDance's Bytespider is the common example — widely reported to ignore directives and to hit hard enough to hurt origin performance. When a crawler will not comply, the only real control is enforcement at the web server. This page throttles Bytespider in Nginx with limit_req, scoped so it never affects Googlebot, and shows how to confirm from the log that the throttle bit. It is the enforcement path for the non-compliant crawlers in the AI crawler and scraper control guide.

Diagnosis: Confirm the Aggression

Rate-limit only what is actually aggressive. Measure Bytespider's request rate — hits per minute at peak:

grep -i 'bytespider' /var/log/nginx/access.log \
  | awk '{print substr($4,2,17)}' | uniq -c | sort -rn | head

Expected Output: per-minute counts, e.g. 640 19/Jun/2026:08:31. Several hundred requests a minute from one crawler is the load worth throttling; a handful is not.

Concept: Throttle by Identity, in Its Own Zone

Nginx limit_req restricts request rate for a key you choose. The safe design is a dedicated zone keyed to a variable that is set only for Bytespider, so the limit applies to it and nothing else. Matching the user-agent with map gives you that variable. Because the zone is separate and the key is empty for everyone else, legitimate traffic — and verified Googlebot — passes untouched, honoring the rule never to throttle a search crawler by a blanket policy from the robots.txt and crawl-rate control guide.

Step-by-Step Fix

Step 1: Map the user-agent to a limit key. In the http block, set a key only when the UA is Bytespider.

# http block
map $http_user_agent $bytespider_key {
    default            "";
    "~*bytespider"     $binary_remote_addr;
}
limit_req_zone $bytespider_key zone=bytespider:10m rate=6r/m;

Expected Output: the config parses. The empty default key means non-Bytespider requests are not counted in the zone at all — Nginx skips the limit when the key is empty.

Step 2: Apply the limit in the location. Allow a small burst so a legitimate short burst is not rejected outright.

# server / location block
location / {
    limit_req zone=bytespider burst=4 nodelay;
    # ... normal handling ...
}

Expected Output: Bytespider requests beyond 6r/m (plus the burst) receive 503; every other client, including Googlebot, is unaffected because their key is empty.

Step 3: Test and reload safely.

nginx -t && systemctl reload nginx

Expected Output: nginx: configuration file /etc/nginx/nginx.conf test is successful, then a graceful reload. Never reload without nginx -t — a bad limit_req directive can stop the worker from starting.

Production Warning: map uses the raw $http_user_agent, which a client controls and can change. This throttles anything calling itself Bytespider, which is the goal, but a scraper can rename itself to evade it. For persistent abuse, escalate to an IP-range block against ByteDance's published ranges, or a CDN rule, rather than relying on the UA string alone.

Edge-Case Handling

  • Rejecting with 429 instead of 503. Add limit_req_status 429; in the location so the response is the more semantically correct "Too Many Requests"; some crawlers back off better on 429.
  • Behind a CDN or load balancer. $binary_remote_addr is the immediate peer — the CDN, not the crawler — so all Bytespider traffic shares one key and is over-throttled. Key on the real client from X-Forwarded-For instead, the same real-IP concern as CDN log analysis for SEO.

Verification

Confirm the throttle is issuing 503s to Bytespider and nobody else:

grep -i 'bytespider' /var/log/nginx/access.log | awk '$9==503' | wc -l
grep -iv 'bytespider' /var/log/nginx/access.log | awk '$9==503 && /Googlebot/' | wc -l

Expected Output: the first count is non-zero (Bytespider is being limited); the second is 0 (no Googlebot request was throttled). A non-zero second count means your key is leaking — recheck the map.

Why Rate-Limiting Beats Outright Blocking

When a crawler ignores robots.txt, the instinct is to block it outright with a 403, but rate-limiting is usually the better response, and understanding why shapes how you handle any non-compliant crawler. An outright block by user-agent invites evasion: a crawler that finds itself blocked can simply change its user-agent string and return unlabeled, at which point it is harder to identify and you have lost the ability to see and manage it. A rate limit, by contrast, keeps the crawler identifiable in your logs while capping the load it can impose — you can still see exactly what it is doing and how much, and it has less incentive to disguise itself because it is not being blocked, just slowed.

This visibility is worth preserving. A rate-limited crawler continues to appear in your logs under its own user-agent, so you can monitor whether the limit is working, whether the crawler's behavior changes, and whether it respects the throttle or escalates — all information you lose the moment you block it into changing its identity. Rate-limiting also fails more gracefully: if you accidentally scope the limit too broadly, a rate limit merely slows some traffic, whereas an over-broad block cuts it off entirely, and the consequences of a mistake are milder. The general principle is to cap a misbehaving crawler's cost while keeping it visible and identifiable, escalating to an outright block only for persistent abuse that a rate limit cannot contain. Choosing the rate limit as the first-line response to a non-compliant crawler — reserving the block for cases where throttling proves insufficient — is what keeps you in control of the crawler rather than pushing it into an evasive game of user-agent whack-a-mole.

Scoping the Limit So It Never Touches Search Crawlers

The cardinal rule of rate-limiting a crawler is that the limit must never affect the search crawlers you depend on, and the entire risk of the technique lives in how narrowly you scope the limit. A rate limit is a blunt instrument if applied broadly — a site-wide request limit throttles everyone, including Googlebot, which can suppress the crawl rate you want high and cost you search visibility. The whole art of safely rate-limiting a bad crawler is confining the limit to precisely that crawler and nothing else, so a search crawler passes through untouched while the target is throttled.

The mechanism for this narrow scoping is a rate-limit key that is set only for the target crawler and empty for everyone else. By keying the limit on a variable that takes a value only when the request matches the target user-agent — and stays empty otherwise — the limit applies exclusively to the target, because a request with an empty key is not counted against the limit at all. This is the design that keeps the limit surgical: Googlebot, whose requests never match the target and so always have an empty key, is never rate-limited, while the target crawler, whose requests set the key, is throttled. The verification that this scoping worked is a log check confirming that the target crawler receives the throttling status codes while no search crawler does — the proof that the limit is confined to its intended target. Getting this scoping exactly right, and verifying it in the logs, is what makes rate-limiting a bad crawler safe: done narrowly and confirmed, it caps the abuser without any risk to the search crawlers that drive your traffic, but done broadly it becomes a self-inflicted crawl-budget problem far worse than the crawler it was meant to contain.

Escalating Beyond the User-Agent

Rate-limiting on the user-agent string handles the crawler that identifies itself, but a determined abuser can change its user-agent to evade the limit, so knowing how to escalate beyond the user-agent is what keeps enforcement effective against a crawler that will not stay labeled. The escalation ladder moves from the most specific and least disruptive control to progressively broader ones as the crawler proves more evasive. The user-agent limit is the first rung; when a crawler renames itself to escape it, the next rung is an IP-range block against the operator's published address ranges, which the crawler cannot change as easily as its user-agent.

The final rung is enforcement at the CDN or WAF, which can apply behavioral rules — rate patterns, fingerprints, challenge responses — that catch an abuser regardless of the user-agent it claims. Each step up the ladder is broader and more powerful but also blunter, so you escalate only as far as the crawler's evasion forces you, staying at the most specific control that works. A compliant crawler needs nothing beyond robots.txt; a non-compliant but honest one is handled by the user-agent rate limit; only a genuinely evasive abuser requires IP-range or CDN enforcement. Matching the enforcement level to how far the crawler goes to evade — and keeping the whole time to the discipline of never catching search crawlers in the net — is what makes escalation effective without becoming heavy-handed, and it is the path forward when the user-agent limit alone proves insufficient against a crawler determined to keep crawling.

Verify the Limit, Every Time

The single non-negotiable step in rate-limiting any crawler is verifying, in the logs, that the limit hit its target and nothing else. A rate limit is invisible until you confirm it: the config may look correct and still fail to throttle the target, or worse, quietly catch a search crawler. So after every rate-limit change, check the logs for the throttling status codes going to the intended crawler and confirm no search crawler received them. This verification is what turns a hopeful configuration into a confirmed control, and it is the safeguard that keeps a mis-scoped limit from silently costing you search visibility. Never consider a rate limit done until the logs prove it throttled exactly what it should and nothing it should not.

Common Mistakes

  • A shared zone or global limit. A site-wide limit_req throttles everyone, including Googlebot, which can suppress crawl rate. Scope the zone to a Bytespider-only key.
  • Keying on the proxy IP behind a CDN. All crawler traffic collapses onto one key and is mis-limited. Key on the real client IP from the forwarded header.

Frequently Asked Questions

Why not just block Bytespider entirely with a 403?
You can — if ($http_user_agent ~* bytespider) { return 403; } — but an outright block invites the scraper to change its user-agent and come back unlabeled, which is harder to see. A rate limit keeps the crawler identifiable in the log while capping its cost; escalate to a block or IP-range deny only if throttling is ignored.

Will rate-limiting Bytespider affect my SEO?
No, as long as the limit is scoped to Bytespider's user-agent and does not touch search crawlers. Bytespider is not a search crawler and sends no ranking signal, so throttling it only protects your bandwidth. The risk is a mis-scoped limit catching Googlebot — the verification step exists to rule that out.

Part of the AI Crawler & Scraper Control from Logs guide.