Parsing User-Agents with Python

The user-agent string is where a log tells you who is crawling, but it is also where the noise lives: Googlebot ships several variants, AI crawlers keep appearing, and scrapers impersonate everyone. This page builds a small Python classifier that turns the raw UA field into a clean category — search crawler, AI crawler, browser, or unknown — so downstream crawl analysis groups traffic correctly. It extends the parsing foundation of the Python logparser setup guide to the identity field.

Diagnosis: Why Naive Grouping Fails

Group by the raw user-agent and you get thousands of near-identical strings that should be one bucket:

awk -F\" '{print $6}' access.log | sort | uniq -c | sort -rn | head -5

Expected Output: several Mozilla/5.0 (compatible; Googlebot/2.1; ...) lines differing only by a version or platform detail — all Googlebot, counted separately. You need to classify on the product token, not the full string.

Concept: Classify on the Product Token

A user-agent is a sequence of product tokens (Googlebot/2.1, GPTBot/1.1) plus comments. Reliable classification keys on the token, not the whole string, and maps a set of known tokens to categories. This is deliberately a coarse classification for grouping crawl traffic — not a full UA-parsing library — because the categories that matter for crawl-budget work are few: is this a search crawler, an AI crawler, or neither, a distinction drawn out in AI crawler and scraper control.

Step-by-Step

Step 1: Define the token-to-category map. A dict of compiled patterns keeps it fast and readable.

import re

CATEGORIES = [
    ("search-crawler", re.compile(r"Googlebot|bingbot|DuckDuckBot|YandexBot|Baiduspider", re.I)),
    ("ai-crawler",     re.compile(r"GPTBot|ClaudeBot|CCBot|PerplexityBot|Bytespider|Google-Extended", re.I)),
    ("browser",        re.compile(r"Mozilla.*(Chrome|Safari|Firefox|Edg)/", re.I)),
]

def classify(ua: str) -> str:
    for name, pat in CATEGORIES:
        if pat.search(ua):
            return name
    return "unknown"

Expected Output: classify("Mozilla/5.0 (compatible; Googlebot/2.1; ...)") returns search-crawler; a GPTBot string returns ai-crawler. Order matters — search and AI patterns are checked before the generic browser pattern, because AI crawlers also send Mozilla/5.0.

Step 2: Tally categories across the log. Extract the quoted UA and count.

import sys, collections
UA = re.compile(r'"[^"]*" "([^"]*)"\s*$')   # referer then user-agent, end of line
counts = collections.Counter()
for line in sys.stdin:
    m = UA.search(line)
    counts[classify(m.group(1) if m else "")] += 1
for cat, n in counts.most_common():
    print(f"{n:8d}  {cat}")

Expected Output:

  318442  browser
   48213  search-crawler
   12450  ai-crawler
    3120  unknown

Step 3: Extract the specific bot token for a breakdown. Pull the product token for finer grouping.

TOKEN = re.compile(r"(Googlebot|bingbot|GPTBot|ClaudeBot|CCBot|Bytespider|PerplexityBot)", re.I)
def bot_token(ua): 
    m = TOKEN.search(ua)
    return m.group(1) if m else None

Expected Output: bot_token returns the canonical crawler name, letting you count Googlebot vs GPTBot vs Bytespider precisely — the same per-bot view as extracting top bot user-agents from logs, now programmatically.

Edge-Case Handling

Empty or malformed UA. A missing user-agent (a -) should classify as unknown, not crash; the classify("") path handles it. Track the unknown rate as a data-quality signal.
Verification is separate. Classification reads the string; it does not prove identity. A "search-crawler" hit can be a spoof, so verify with reverse DNS before trusting counts, per detecting fake Googlebot traffic.

Verification

Confirm the category counts sum to the total parsed lines, so no line is silently dropped:

assert sum(counts.values()) == parsed_lines, "category counts do not reconcile"

Expected Output: no assertion error. A mismatch means the UA regex failed on some lines — inspect them; a custom log format may place the UA in a different position.

Why Classification Beats Full Parsing for Crawl Work

There is a whole category of user-agent parsing libraries that decompose a UA string into browser, version, engine, operating system, and device, and it is tempting to reach for one. For crawl-budget analysis, that is the wrong tool, and understanding why clarifies what your classifier actually needs to do. Full UA parsing is built for analytics about human visitors — which browsers and devices your audience uses — a question that is irrelevant when your subject is bots. What you need for crawl work is coarse and categorical: is this request from a search crawler, an AI crawler, a browser, or something unknown, and if it is a bot, which one. A token-based classifier answers exactly that, and nothing more, which makes it faster, dependency-free, and easier to keep current.

The efficiency difference compounds at log scale. A full UA parser does substantial work per string — matching against large rule sets to extract every attribute — which is wasted effort when you only care about a handful of bot tokens. A classifier that checks a short list of crawler product tokens and falls back to a browser-versus-unknown decision does a fraction of the work and produces exactly the categories crawl analysis groups by. Just as importantly, keeping the classification logic small and in your own code means you can add a newly-appeared AI crawler the moment it shows up in your logs, rather than waiting for a library to update its rules. For crawl-budget work, the right abstraction is a small, maintainable classifier tuned to the bots you care about, not a general-purpose parser tuned to the browsers you do not.

Keeping the Classifier Accurate Over Time

A crawler classifier is only useful if it stays current, and the crawler landscape changes faster than any static list — new AI crawlers launch, operators rename tokens, and search engines add specialized fetchers. A classifier that was accurate six months ago silently miscategorizes today's traffic, folding new crawlers into the unknown bucket where they escape your analysis. The maintenance discipline is to treat the unknown category not as a dumping ground but as a worklist: periodically review the highest-volume unknown user-agents and decide whether each is a new crawler that deserves its own category.

# Surface the top unknown user-agents — candidates to add to the classifier
import collections
unknown = collections.Counter()
for line in log_lines:
    ua = extract_ua(line)
    if classify(ua) == "unknown":
        unknown[ua] += 1
for ua, n in unknown.most_common(15):
    print(n, ua[:90])

Expected Output: a ranked list of the user-agents your classifier does not recognize, where a new AI crawler appearing in volume stands out as a candidate to add. Reviewing this monthly and folding recognized new crawlers into the classifier's token map keeps the categorization accurate as the landscape shifts, so your crawl analysis continues to see AI crawlers as AI crawlers rather than losing them in the unknown pile. This review loop is what turns a static classifier into a living one, and it costs only a few minutes against the payoff of never being blind to a new bot that started crawling your site last week.

Integrating Classification Into the Parse Pipeline

The classifier is most valuable when it is not a standalone step but a field that every parsed log record carries, so downstream analysis can group by crawler category without re-classifying. Folding the classification into the parser means each record emerges tagged — search-crawler, ai-crawler, browser, or unknown, plus the specific bot token where one matched — and every report, aggregation, and filter can key on those tags directly. This turns crawler categorization from something each analysis re-derives into a stable dimension of the parsed data.

The integration is a small addition to the parse function: after extracting the user-agent field, run it through the classifier and attach both the category and the specific bot token to the record. From there, a status-distribution report can filter to category == "search-crawler" in one step, a bandwidth report can group by bot token, and an AI-crawler analysis can select category == "ai-crawler" without touching the raw user-agent string again. Because the classification happens once at parse time and travels with the record, the whole downstream analysis becomes cleaner and faster, and the categorization stays consistent across every report rather than each one applying its own slightly-different matching. This is the same principle that makes any derived field worth computing once in the parser rather than repeatedly in each consumer: the parse is where identity is established, and every analysis benefits from receiving records that already know what they are.

Classification as a Shared Building Block

The user-agent classifier is most valuable not as a standalone script but as a shared building block that many analyses import, and structuring it that way is what keeps crawler categorization consistent across everything you build. When each analysis re-implements its own crawler matching, the analyses drift — one counts a crawler the classifier would categorize differently, one omits a new AI crawler another includes — and comparing their results becomes a hunt for which one matched crawlers which way. A single shared classifier that every analysis imports eliminates the drift by making them all categorize crawlers identically.

The shared classifier also concentrates the maintenance in one place. As new crawlers appear and tokens change, the classifier needs updating, and when it lives in one shared module, an update propagates to every analysis at once; when it is duplicated across scripts, each copy must be found and updated separately, and some will be missed. So the classifier becomes the single, maintained, trusted definition of what each crawler is, which every crawl analysis builds on — the same principle that makes the reusable log parser worth building. Packaging the crawler classification as a shared, importable component with its token map in one place is what keeps crawler categorization coherent as your set of analyses grows and the crawler landscape shifts, turning it from a snippet each script re-derives into a common vocabulary every analysis speaks.

The Classifier as a Living Component

A crawler classifier is a living component, not a finished artifact, because the crawler landscape it categorizes keeps changing. New AI crawlers appear, tokens are renamed, and search engines add specialized fetchers, so a classifier that was complete six months ago now miscategorizes the newcomers into its unknown bucket. Keeping it accurate means treating the unknown category as a standing worklist — reviewing the top unrecognized user-agents periodically and folding genuine new crawlers into the token map. This small, recurring maintenance is what keeps the classifier matched to a moving reality, so your crawl analysis continues to see AI crawlers as AI crawlers rather than losing them among the unclassified. A classifier maintained this way stays useful indefinitely; one left static silently decays as the landscape moves past it.

Common Mistakes

Checking the browser pattern first. AI and search crawlers also send Mozilla/5.0, so a browser-first order misclassifies them. Check specific crawler tokens before the generic browser pattern.
Treating classification as verification. The string is spoofable. Classify for grouping, verify by IP for anything you act on.

Frequently Asked Questions

Should I use a full user-agent parsing library instead?
For crawl-budget work, no — a small token map is faster, dependency-free, and easier to keep current as new AI crawlers appear. Full UA libraries excel at fine-grained browser and device detection, which is not the question when you are grouping crawler traffic.

How do I keep the crawler list current?
Keep the category patterns in one place (the CATEGORIES list) and add tokens as new crawlers show up in your unknown bucket. Periodically review the top unknown user-agents to catch a new bot before it becomes a large uncategorized slice.

Handling malformed log lines in Python — make the parser robust to bad input.
Identifying search engine bots — verify the crawlers your classifier groups.

Part of the Python Logparser Setup guide.