Blocking GPTBot and CCBot in robots.txt

The task is narrow and common: stop OpenAI's GPTBot and Common Crawl's CCBot from crawling your site for AI training data, while leaving search crawlers completely unaffected. Both are compliant crawlers that read robots.txt, so a few correct lines do the job — the risk is not the syntax but accidentally catching Googlebot or writing a rule the crawler never actually reads. This is the enforcement detail behind the parent AI crawler and scraper control guide.

Diagnosis: Confirm They Are Crawling You

First prove there is something to block. Count GPTBot and CCBot hits in the log:

grep -icE 'GPTBot|CCBot' /var/log/nginx/access.log
grep -iEo 'GPTBot|CCBot' /var/log/nginx/access.log | sort | uniq -c

Expected Output: per-token counts, e.g. 5120 GPTBot and 1440 CCBot. Zero means they are not visiting and there is nothing to do yet.

Concept: How robots.txt Groups Work

robots.txt matches the most specific User-agent group to each crawler, and each named group is independent. A group for GPTBot governs only GPTBot; it has no effect on Googlebot. This is what lets you block AI training crawlers while search crawlers, which match a different (or the wildcard) group, retain full access. The token names are case-insensitive and must match the crawler's documented product token exactly.

Step-by-Step Fix

Step 1: Add named groups for each AI crawler. Place them in the root robots.txt.

# robots.txt
User-agent: GPTBot
Disallow: /

User-agent: CCBot
Disallow: /

Expected Output: the file, served at https://example.com/robots.txt, now contains an explicit block for each token. Do not add these under a shared User-agent: * line — keep each in its own group so nothing else is affected.

Step 2: Confirm Googlebot is untouched. Your search-crawler access should come from the absence of a restrictive rule. Verify there is no over-broad block above these groups:

curl -s https://example.com/robots.txt | grep -iA2 'User-agent'

Expected Output: a User-agent: GPTBot group and a User-agent: CCBot group each followed by Disallow: /, and no User-agent: * group that disallows the whole site. Googlebot, matching no restrictive group, keeps full access.

Step 3: Allow selectively if you prefer. To let them read some paths but not others, replace the blanket disallow:

User-agent: GPTBot
Allow: /blog/
Disallow: /

Expected Output: GPTBot may fetch /blog/ but nothing else. Order does not matter to compliant crawlers; they apply the most specific matching rule per URL.

Edge-Case Handling

  • ChatGPT-User vs GPTBot. GPTBot is the training crawler; ChatGPT-User and OAI-SearchBot are user- and search-triggered fetches. Block them separately if you want to stop those too — a GPTBot rule does not cover them.
  • CDN-served robots.txt. If a CDN caches robots.txt, purge the cache after editing or crawlers keep reading the old file for the cache lifetime.

Verification

The only proof that matters is the log after a crawl cycle. Watch the daily counts fall:

grep -i 'gptbot' /var/log/nginx/access.log | awk '{split($4,d,":"); print substr(d[1],2)}' | sort | uniq -c

Expected Output: GPTBot's daily count dropping to zero within a day or two of the change. Compliant crawlers cache robots.txt briefly, so allow a cycle before concluding anything.

Why These Two Crawlers Get Blocked Most Often

GPTBot and CCBot draw more blocking decisions than any other AI crawlers, and understanding why clarifies whether blocking them is right for your site. GPTBot is OpenAI's training crawler, gathering content to train and update the models behind ChatGPT — for many publishers, the most direct case of their content being used to build a product they do not control and are not compensated for. CCBot belongs to Common Crawl, whose openly-published web archive is a primary training source for a wide range of AI models, so blocking CCBot cuts off not one model but a whole ecosystem that draws from the Common Crawl corpus. Together they represent the clearest, highest-leverage points at which a site can opt out of feeding AI training.

That leverage is exactly why the decision deserves thought rather than reflex. Blocking GPTBot stops OpenAI from ingesting your new content going forward, and blocking CCBot removes you from the archive many models train on — but neither retroactively removes content already gathered, and neither affects your search visibility, since both are entirely separate from search crawlers. The trade is between keeping your content out of training corpora and any benefit you might get from being represented in the answers those models produce. There is no universally right choice: a publisher protecting proprietary content reasonably blocks both, while a site that wants to be the authoritative source in AI answers might allow them. What makes GPTBot and CCBot the usual subjects of blocking is simply that they are the two crawlers where the opt-out is most consequential, which is why getting the robots.txt directives exactly right for them matters.

Confirming the Block Actually Took Effect

Writing the robots.txt directive is only half the job; the other half is confirming from the logs that the crawlers actually stopped, because a block you never verified is a block you cannot trust. Compliant crawlers cache robots.txt for a period before re-reading it, so the effect is not instant — you should expect the crawler's hit count to decline over the day or two following the change as it picks up the new file, not to drop to zero the moment you save it. Watching that decline in the logs is what turns "I added the directive" into "the directive worked."

The verification is a simple trend: capture the crawler's daily hit count before the change as a baseline, apply the block, and track the count over the following days. A compliant crawler like GPTBot or CCBot should trend toward zero within a crawl cycle; if it does not, either the robots.txt is not being served correctly (wrong location, a 404, a caching CDN serving a stale copy) or the traffic claiming to be that crawler is spoofed and never honored robots.txt in the first place. Distinguishing these matters, because they call for different fixes — a serving problem needs a robots.txt correction, while spoofed traffic needs enforcement at the server or CDN. Making the log-verified decline a required step of any robots.txt block, rather than assuming the directive worked because the syntax is correct, is what keeps your crawler-control decisions grounded in what actually happened rather than what you intended.

Handling the Crawlers That Do Not Comply

The robots.txt approach works only for crawlers that honor it, and while GPTBot and CCBot both document compliance, the broader lesson is that robots.txt is a request, not an enforcement mechanism. A crawler that chooses to ignore your directives — as some scrapers and less-scrupulous bots do — will keep fetching regardless of what your robots.txt says, and no amount of correct syntax changes that. Recognizing this boundary is what tells you when to escalate from robots.txt to actual enforcement.

The signal that you have crossed that boundary is a crawler whose hit count does not decline after you block it, verified over a full crawl cycle so you are not mistaking caching lag for non-compliance. Once you confirm a crawler is ignoring robots.txt, the tool changes: you enforce at the web server with a rate limit or a conditional block on the user-agent, or at the CDN or WAF for persistent abuse, as the parent AI crawler and scraper control guide describes. The escalation ladder — robots.txt for the compliant, server rules for the non-compliant, CDN for the evasive — matches the control to the crawler's demonstrated behavior. GPTBot and CCBot sit firmly on the first rung because they comply, which is why robots.txt is the right and sufficient tool for them; the value of knowing the ladder exists is recognizing quickly when a different crawler has stepped off it and needs a stronger response.

Revisiting the Block as the Landscape Changes

A robots.txt block on GPTBot and CCBot is not a set-and-forget decision, because the AI-crawler landscape and your own strategy both change, so revisiting the block periodically is what keeps it aligned with your current interests rather than a stale choice made once. New AI crawlers appear that the existing block does not cover; the operators of GPTBot and CCBot may change how their crawlers behave or what blocking them implies; and your own view of whether you want your content in AI training or answers may shift as the technology and its role in discovery evolve.

Building a periodic review into your process keeps the policy current. The review asks whether new AI crawlers have appeared that warrant their own directives, whether the crawlers you block still behave as expected, and whether your strategic stance on AI crawling still holds given how the ecosystem has moved. Because AI crawling is a fast-changing area, a block that made sense a year ago may be incomplete or misaligned now, and only a deliberate review catches that drift. The logs feed the review, showing which AI crawlers are actually hitting you and how that traffic is trending, so the decision to keep, adjust, or extend the block rests on current evidence. Treating the GPTBot and CCBot block as a living policy to revisit as the landscape changes — rather than a one-time robots.txt edit — is what keeps your AI-crawler control matched to a fast-moving reality, and it fits the same recurring-review discipline that keeps every crawler-control decision current.

A Decision to Make Deliberately

Blocking GPTBot and CCBot is a genuine decision, not a default, and making it deliberately — informed by your own view of AI crawling and your own log evidence — is what keeps it aligned with your interests. The block is easy to apply, but whether to apply it depends on your content, your business model, and your stance on your content feeding AI training and answers. The logs inform the decision by showing what these crawlers actually cost you and how that is trending, but the choice itself is yours to make with intent rather than reflexively. Deciding to block, allow, or selectively permit these crawlers as a considered choice, revisited as the landscape shifts, is what keeps your AI-crawler policy a reflection of your actual interests rather than an unexamined default.

Common Mistakes

  • Putting AI tokens under User-agent: *. That can disallow the whole site for everyone, including search crawlers. Give each AI crawler its own group.
  • Serving robots.txt from the wrong place. It must be at the domain root and return 200. A 404 or a subdirectory copy is never read.

Frequently Asked Questions

Will blocking GPTBot remove my site from ChatGPT?
Blocking GPTBot stops OpenAI's training crawler from fetching new content, but it does not retroactively remove content already gathered, and it does not affect ChatGPT-User browsing fetches unless you block that token too. It has no effect on your Google or Bing search presence.

Do GPTBot and CCBot actually obey robots.txt?
Yes — both are documented as respecting robots.txt. You should still verify from the log after a crawl cycle, because a spoofed user-agent from an unrelated IP will not comply; that is a scraper impersonating them, handled with rate limits rather than robots.txt.

Part of the AI Crawler & Scraper Control from Logs guide.