AI Crawler & Scraper Control from Logs

A new class of bot now shows up in access logs: AI crawlers gathering training and retrieval data — OpenAI's GPTBot, Anthropic's ClaudeBot, Common Crawl's CCBot, PerplexityBot, and ByteDance's Bytespider, among others. They are not search crawlers, they do not send referral traffic, and some hit hard enough to distort your crawl statistics and load. This guide identifies them in the log, measures what they actually cost you, and steers or throttles them with the same precision you would apply to any bot — without ever catching Googlebot in the net. It is the AI-specific extension of the verification work in identifying search engine bots.

The decision is yours to make deliberately: allowing AI crawlers may surface your content in AI answers; blocking them protects bandwidth and content. Either way, the log is where you see the real trade, and robots.txt plus rate limits are how you enforce it.

What you will accomplish:

  • Recognize the major AI-crawler user-agents and separate them from search crawlers.
  • Quantify AI-crawler request volume, bandwidth, and status mix from the log.
  • Steer compliant AI crawlers with robots.txt tokens like GPTBot and Google-Extended.
  • Rate-limit or block aggressive, non-compliant scrapers at the edge without touching Googlebot.

The matrix below maps the common AI crawlers to their user-agent token, operator, robots.txt control, and whether they honor it.

Common AI crawlers and their controls Rows for GPTBot by OpenAI, ClaudeBot by Anthropic, CCBot by Common Crawl, Google-Extended by Google, PerplexityBot by Perplexity, and Bytespider by ByteDance, each showing whether it declares a documented user-agent and honors robots.txt. AI crawlers in your logs User-agent token Operator Honors robots.txt? GPTBotOpenAIYes — documented ClaudeBotAnthropicYes — documented CCBotCommon CrawlYes — documented Google-ExtendedGoogle (AI training)Yes — robots token PerplexityBotPerplexityDocumented; verify BytespiderByteDanceOften ignores — rate-limit

Prerequisites

You need access logs in combined format (so the user-agent is present), the ability to edit your public robots.txt, and — for enforcement — control of your web server or CDN config to apply rate limits. As always, the analysis is only trustworthy against verified traffic, so keep the reverse-DNS techniques from detecting fake Googlebot traffic in mind: an aggressive scraper may spoof an AI user-agent just as readily as a search one. Confirm your logs actually capture the user-agent:

awk -F\" '{print $6}' /var/log/nginx/access.log | grep -iE 'bot|crawler|spider' | sort | uniq -c | sort -rn | head

Expected Output: a ranked list of bot user-agents seen, e.g. 48213 Mozilla/5.0 (compatible; Googlebot/2.1; ...), 5120 ... GPTBot/1.1 .... If the user-agent column is empty, switch to the combined format per the Apache vs Nginx log formats guide before proceeding.

Environment Setup: Fingerprint the AI Crawlers

Build one regex that matches the documented AI-crawler tokens, then confirm which ones appear in your logs and at what volume.

AI='GPTBot|ChatGPT-User|OAI-SearchBot|ClaudeBot|Claude-Web|anthropic-ai|CCBot|Google-Extended|PerplexityBot|Bytespider|Amazonbot|Applebot-Extended'
grep -iEo "$AI" /var/log/nginx/access.log | sort | uniq -c | sort -rn

Expected Output:

   5120 GPTBot
   3880 Bytespider
   2010 ClaudeBot
   1440 CCBot
    260 PerplexityBot

Verification command. Google-Extended is a special case: it is a robots.txt token that governs whether Google may use your content for AI training, but it is not a distinct user-agent in the logs — Googlebot fetches on Google's behalf. So do not expect to see Google-Extended as a UA; control it in robots.txt only. Note also that ChatGPT-User and OAI-SearchBot are user-triggered fetches, distinct from the training crawler GPTBot.

Pipeline / Agent Configuration: Steer Compliant Crawlers with robots.txt

Compliant AI crawlers read robots.txt and obey per-user-agent rules. Add explicit groups; each token is matched independently, so a rule for GPTBot does not affect Googlebot.

# robots.txt — allow search, control AI training crawlers
User-agent: GPTBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: ClaudeBot
Disallow: /

# Googlebot and Bingbot are untouched — no rule here means full access

Expected Output: on the next crawl cycle, verified GPTBot/CCBot/ClaudeBot hits in the log fall toward zero while Googlebot volume is unchanged. Allow selectively by replacing Disallow: / with specific paths.

Production Warning: Google-Extended controls only Gemini/Vertex AI training use of your content; it does not affect Google Search crawling or ranking, and disallowing it does not remove you from Search. Conversely, blocking Googlebot to stop AI features would deindex you — never conflate the two tokens.

Step 2: Give AI crawlers time and re-measure. robots.txt is cached; a crawler may take a day or more to pick up the change. Confirm compliance by re-running the fingerprint after a crawl cycle and checking the token counts dropped.

Parsing Logic & Field Mapping: Measure the Real Cost

Volume alone understates the impact; bandwidth and status mix tell the fuller story. Sum bytes and count statuses per AI crawler.

awk '/GPTBot|Bytespider|ClaudeBot|CCBot/ {
  match($0, /(GPTBot|Bytespider|ClaudeBot|CCBot)/, m);
  bytes[m[1]] += $10; hits[m[1]]++
} END { for (b in bytes) printf "%-12s %8d hits  %10.1f MB\n", b, hits[b], bytes[b]/1048576 }' \
  /var/log/nginx/access.log

Expected Output:

GPTBot        5120 hits      842.3 MB
Bytespider    3880 hits      611.7 MB
ClaudeBot     2010 hits      190.4 MB
CCBot         1440 hits      142.8 MB

SEO-relevant field callout. AI-crawler traffic does not consume Google's crawl budget — that budget is per search engine — but it does consume your server capacity, and a slow origin under scraper load can throttle the effective crawl rate of the search bots you do want, the coupling explained in the Crawl Budget Optimization & Bot Management guide. So the bandwidth column, not the SEO column, is where AI crawlers hurt.

Validation & Troubleshooting

Health check: is a bot honoring robots.txt? After disallowing a token, its hits should trend to zero. If they do not, the bot is either lagging on the cache or ignoring the rule.

grep -i 'gptbot' /var/log/nginx/access.log | awk '{split($4,d,":"); print substr(d[1],2)}' | sort | uniq -c

Expected Output: daily GPTBot counts declining after the robots.txt change. A flat line after several days means non-compliance — escalate to a block.

Recovery recipes for named failure modes:

  • Bytespider ignores robots.txt. It is widely reported to disregard directives. Enforce at the edge with a rate limit or a user-agent block rather than relying on robots.txt.
  • A spoofed AI user-agent. A scraper claiming to be GPTBot from a non-OpenAI IP is a fake. OpenAI publishes GPTBot IP ranges; verify against them, or simply rate-limit by IP behavior as with any unverified bot.
  • robots.txt change had no effect. Confirm the file is served at the domain root with 200 and correct syntax (curl -s https://example.com/robots.txt); a 404 or a subdirectory placement means no crawler ever read it.
  • Blocked the wrong bot. Re-read your groups: an over-broad User-agent: * Disallow above a specific group can capture search crawlers. Keep AI rules in their own named groups.

Distinguishing Training Crawlers from Retrieval Fetchers

A crucial nuance that shapes your whole policy: the AI bots in your logs are not all doing the same thing. Training crawlers (GPTBot, CCBot, Google-Extended, ClaudeBot) gather content to train or update models — a bulk, background activity. Retrieval fetchers (ChatGPT-User, OAI-SearchBot, PerplexityBot on a live query) fetch a page in real time to answer a specific user's question — closer to a referral than a scrape. Blocking the first keeps your content out of training sets; blocking the second removes you from AI answers that might actually send interested users your way.

# Separate background training crawl from user-triggered retrieval fetches
awk -F\" '{ua=$6}
  ua ~ /GPTBot|CCBot|ClaudeBot|Google-Extended/ {train++}
  ua ~ /ChatGPT-User|OAI-SearchBot|PerplexityBot/ {retrieve++}
  END {print "training:", train+0, " retrieval:", retrieve+0}' access.log

Expected Output:

training: 8200  retrieval: 640

A high training count with low retrieval means AI systems are ingesting your content but rarely surfacing it live; a rising retrieval count means your pages are being cited in AI answers, which some sites actively want. Deciding your policy per-category rather than blocking "AI bots" wholesale is what separates a considered stance from a blunt one — you might block training while allowing retrieval, or vice versa, and the log is the only place you can see which is which.

Measuring the SEO-Adjacent Value of AI Referrals

The counterargument to blocking AI crawlers is that AI assistants increasingly cite sources, and those citations can send real traffic — a channel that behaves like search referral but does not show up in your rank tracking. Before you block, it is worth measuring whether AI retrieval fetches correlate with actual referral visits, because blocking a crawler that drives citations is a different trade than blocking one that only consumes bandwidth.

# Retrieval fetches often precede a human visit from the same AI product's referrer
grep -iE 'ChatGPT-User|PerplexityBot' access.log | awk '{print $1}' | sort -u > ai_fetch_ips.txt
grep -iE 'referer.*(perplexity|openai|chatgpt)' access.log | wc -l

Expected Output: a count of human visits arriving with an AI-product referrer, which you can weigh against the bandwidth the retrieval fetchers consumed. If retrieval fetches are followed by meaningful referral traffic, that fetcher is earning its bandwidth and blocking it costs you visits; if not, it is pure cost. This measurement reframes AI-crawler control from a binary block/allow into a value calculation — the same "cost versus benefit in the logs" discipline the rest of the Crawl Budget Optimization & Bot Management guide applies to search crawlers.

Keeping the Crawler List Current

The AI-crawler landscape changes fast — new bots appear, user-agent tokens change, and operators add or rename crawlers — so a static block list silently goes stale. The maintenance discipline is to periodically review your unknown/uncategorized bot traffic for new high-volume agents and add them to your fingerprint and policy before they become a large unmanaged slice.

# Surface unrecognized high-volume bots — candidates for your next policy update
awk -F\" '{print $6}' access.log \
  | grep -iE 'bot|crawler|spider|gpt|ai|llm' \
  | grep -viE 'Googlebot|bingbot|GPTBot|ClaudeBot|CCBot|Bytespider|PerplexityBot|Applebot|Amazonbot' \
  | sort | uniq -c | sort -rn | head

Expected Output: a ranked list of bot user-agents your current rules do not recognize, e.g. a newly-launched AI crawler appearing in volume. Reviewing this monthly and folding new agents into your robots.txt and rate-limit rules keeps the policy current as the ecosystem shifts, so a new training crawler does not quietly ingest your whole site for weeks before you notice it. Treating the unknown-bot bucket as a worklist, rather than ignoring it, is what keeps AI-crawler control from decaying the moment the landscape moves.

The Strategic Question: To Allow or Block

Underneath the mechanics sits a genuine strategic decision that every site owner now faces, and the logs inform it without making it for you. Allowing AI training crawlers means your content may shape how AI systems answer questions in your domain — potentially building authority and, as AI assistants increasingly cite sources, driving referral traffic. Blocking them protects bandwidth, keeps your content out of training corpora you did not consent to, and preserves whatever competitive value your content holds. Neither choice is universally correct; it depends on your content, your business model, and your view of how AI-mediated discovery will evolve.

What the logs give you is the evidence to decide with, rather than on instinct. They show exactly which AI crawlers visit, how much they cost in bandwidth, whether their retrieval fetches correlate with referral visits, and how that traffic is trending month over month. A publisher whose revenue depends on direct visits may reasonably block training crawlers while allowing retrieval fetchers that cite and link back. A documentation site that benefits from being the authoritative answer everywhere might allow all of them. A bandwidth-constrained site facing an aggressive scraper simply enforces limits regardless of the strategic debate. The point is that the decision should be revisited as the landscape shifts — new crawlers, changed citation behavior, new evidence in your own logs — rather than set once and forgotten. Treating AI-crawler policy as a living decision informed by your own traffic data, not a one-time robots.txt edit, is what keeps it aligned with your actual interests as both the technology and your content strategy evolve.

Layering Controls: robots.txt, Server Rules, and CDN

Effective AI-crawler control is rarely a single mechanism; it is a layered defense matched to how compliant each crawler is. The first layer is robots.txt, which cleanly handles the well-behaved majority — GPTBot, CCBot, ClaudeBot, and Google-Extended all document that they honor it, so a few directives keep them out at zero cost and with no false positives on search crawlers. This layer does the bulk of the work for the crawlers that play by the rules, and it should be your first and primary control precisely because it is surgical and safe.

The second layer is server-level enforcement for the crawlers that ignore robots.txt. A rate limit or a conditional block at the web server, scoped to a specific non-compliant user-agent, caps the load a crawler like Bytespider can impose when it disregards your directives — the technique detailed in rate-limiting Bytespider in Nginx. The third layer, for persistent or evasive abuse, is the CDN or WAF, which can enforce rules closer to the client, match on IP ranges as well as user-agents, and absorb the load before it reaches your origin at all. The layers escalate with the crawler's behavior: robots.txt for the compliant, server rules for the non-compliant, CDN or WAF for the evasive. Matching the control to the crawler's demonstrated behavior — rather than applying one heavy mechanism to everything — is what keeps enforcement both effective against bad actors and safe for the legitimate search crawlers you never want to touch. Verifying each layer's effect in the logs, as the rest of this guide stresses, is how you confirm the escalation actually worked rather than assuming it did.

The Robots.txt Order and Precedence Trap

A subtle but common failure when managing many crawler groups in robots.txt is precedence — how a crawler chooses which group of rules applies to it. Each crawler obeys exactly one group: the one whose User-agent line most specifically matches its token, falling back to the User-agent: * wildcard group only if no named group matches. This means an over-broad wildcard Disallow and a set of named AI-crawler groups can interact in ways that surprise you, and the interaction is exactly where sites accidentally block search crawlers or fail to block AI ones.

The safe structure keeps each AI crawler in its own explicit named group and never relies on the wildcard group to do AI-specific work. If you put Disallow: / under User-agent: * intending to block AI bots, you also block every crawler that lacks a more specific group — potentially search crawlers, depending on what named groups exist — which is how a well-meant AI block deindexes a site. Conversely, a named User-agent: GPTBot group with its own rules cleanly governs only GPTBot, leaving search crawlers to match their own groups or the wildcard as intended. The discipline is to give each crawler you want to control its own named group with complete rules, treat the wildcard group as the policy for "everyone else" rather than a place to sneak in AI blocks, and verify the result in the logs by confirming each crawler's volume changed in the direction you intended and no search crawler's did. Getting the group structure right is what separates a robots.txt that does exactly what you meant from one that silently blocks the wrong bots.

Monitoring the Effect of Every Control You Apply

The discipline that ties all of this together is verification: every control you apply — a robots.txt block, a rate limit, a CDN rule — must be confirmed in the logs, because the alternative is believing a control worked when it did not. A robots.txt Disallow on a crawler that turns out to ignore it, a rate limit scoped to the wrong user-agent, a CDN rule that never matched — each fails silently, and only the log tells you. The habit worth building is that no crawler-control change is "done" when the config ships; it is done when the logs show the crawler's behavior changed in the direction you intended, and no other crawler's did.

Concretely, that means capturing a baseline of the target crawler's daily volume before the change, applying the control, and watching the volume over the following days — a compliant crawler's hits should fall toward zero within a crawl cycle, a rate-limited one should show the throttling status codes, and every search crawler's volume should be unchanged. A control that produces no change in the log is a control that did not work, whatever the config says, and it needs escalation to the next enforcement layer. A control that changed the wrong crawler's volume is a scoping error that needs immediate correction before it costs you search visibility. Building this verify-in-the-logs step into every crawler-control change is what separates managing AI crawlers by evidence from managing them by hope — and it is the same measurement discipline the whole Crawl Budget Optimization & Bot Management practice rests on, applied to the newest and fastest-changing category of bot in your logs.

Documenting Your AI-Crawler Policy

A crawler-control policy that lives only in a robots.txt file and a few server rules is fragile — the reasoning behind each decision is lost, and the next person to touch it cannot tell an intentional block from an accident. Documenting why each AI crawler is allowed, blocked, or throttled, alongside the log evidence that informed the choice, turns a scattered set of rules into a maintainable policy. Record the date, the crawler, the decision, the measured cost or benefit that drove it, and the log signal you will watch to confirm it still holds. When the landscape shifts — a new crawler, a changed citation behavior, a bandwidth spike — the documented policy tells you what you decided and why, so revisiting it is a considered update rather than a fresh guess. This lightweight record is what keeps AI-crawler management coherent as both the ecosystem and your team change over time.

Common Mistakes

  • Confusing Google-Extended with Googlebot. The former governs AI-training use only; blocking the latter deindexes you. They are different tokens with very different consequences.
  • Relying on robots.txt for non-compliant scrapers. Bots like Bytespider often ignore it. For those, only an edge rate limit or block actually stops the load.
  • Blocking by user-agent alone. A spoofed UA slips a block while a real crawler sharing a string gets caught. Verify by IP where it matters.
  • Ignoring bandwidth. AI crawlers rarely hurt SEO directly, but their bandwidth can slow the origin enough to depress search-crawler rates. Measure bytes, not just hits.

Frequently Asked Questions

Does blocking AI crawlers hurt my search rankings?
No. GPTBot, CCBot, ClaudeBot, and Google-Extended are separate from the search crawlers (Googlebot, Bingbot). Disallowing the AI tokens in robots.txt does not affect crawling, indexing, or ranking in search — it only governs whether your content feeds AI training and retrieval. Blocking Googlebot, by contrast, would deindex you.

Why does Bytespider keep hitting my site after I disallowed it?
Bytespider is widely reported to ignore robots.txt. When a crawler does not comply, robots.txt cannot stop it — you must enforce at the web server or CDN with a rate limit or a user-agent/IP block, as covered in rate-limiting the aggressive scrapers.

Do AI crawlers use up my Google crawl budget?
No. Crawl budget is allocated per search engine, so GPTBot or CCBot traffic does not subtract from Googlebot's budget. The indirect risk is capacity: heavy AI-crawler load can slow your origin, and a slow origin reduces the rate at which search crawlers fetch.

Should I allow AI crawlers at all?
It is a content and bandwidth decision, not an SEO one. Allowing them can surface your content in AI answers and assistants; blocking them conserves bandwidth and keeps content out of training sets. Decide per crawler, measure the cost in the logs, and enforce with robots.txt for compliant bots and rate limits for the rest.

Part of the Crawl Budget Optimization & Bot Management series.