Counting Unique URLs Crawled Per Day with awk

Total crawl hits per day is a noisy number — a crawler can hammer ten URLs a thousand times and look busy while ignoring your catalogue. The metric that actually tracks coverage is distinct URLs fetched per day. This page is the one-line awk recipe that computes it, and the interpretation that makes it useful. It is a coverage-focused companion to the broader recipes in CLI one-liners for quick audits.

Diagnosis: Hits Are Not Coverage

Compare total hits to unique URLs for one day and the gap tells the story:

grep '19/Jun/2026' access.log | grep -c 'Googlebot'
grep '19/Jun/2026' access.log | awk '/Googlebot/{print $7}' | sort -u | wc -l

Expected Output: two numbers — e.g. 6890 total hits but only 2130 distinct URLs. A large ratio means the crawler is re-fetching a small set rather than exploring; a healthy large site wants distinct-URL counts that grow toward its indexable page count.

Concept: Counting Distinct Keys per Group

Counting unique values within each day is a grouped-distinct-count — awk does it by keying a nested structure on day plus url and counting how many unique urls each day accumulates. Because awk holds the seen-set in memory, it needs one pass over the log with no external sort, which is what makes it fast on large files, the same single-pass efficiency behind finding top 404 URLs with awk.

Step-by-Step

Step 1: The one-liner. Extract the day from the [dd/Mon/yyyy:...] timestamp (field 4) and the path (field 7), and count distinct paths per day.

awk '/Googlebot/ {
  split($4, t, ":"); day = substr(t[1], 2);
  if (!seen[day, $7]++) unique[day]++
} END { for (d in unique) print unique[d], d }' access.log | sort -k2

Expected Output:

2130 18/Jun/2026
2384 19/Jun/2026
2201 20/Jun/2026

Step 2: Strip query strings for canonical coverage. Parameter variants inflate the unique count with non-canonical URLs. Fold them out.

awk '/Googlebot/ {
  split($4, t, ":"); day = substr(t[1], 2);
  u = $7; sub(/\?.*/, "", u);
  if (!seen[day, u]++) unique[day]++
} END { for (d in unique) print unique[d], d }' access.log | sort -k2

Expected Output: lower, cleaner per-day counts reflecting distinct canonical paths — the number that maps to real coverage.

Edge-Case Handling

  • Verify the crawler first. Googlebot in the user-agent includes spoofs; a scraper crawling thousands of URLs inflates coverage falsely. Filter to verified hits using verifying Googlebot with reverse DNS for an accurate figure.
  • Multi-day files. The recipe already groups by day, so a log spanning several days reports each separately — no need to pre-split.

Verification

Cross-check one day's unique count against an independent sort-based count:

grep '19/Jun/2026' access.log | awk '/Googlebot/{u=$7; sub(/\?.*/,"",u); print u}' | sort -u | wc -l

Expected Output: the same number the awk one-liner reported for 19/Jun/2026. A mismatch means the day-extraction split differs — align the substring.

Coverage Ratio: Distinct URLs Against Your Site Size

A distinct-URL count is more meaningful as a ratio than an absolute number. If your site has 5,000 indexable pages and crawlers touch 2,000 distinct URLs a day, your daily coverage is 40% — a figure you can track over time and compare against the total. The ratio, not the raw count, tells you whether crawlers are reaching most of your site or churning a fraction of it.

# Daily distinct-URL count as a fraction of your indexable page count
site_pages=5000
awk '/Googlebot/ {u=$7; sub(/\?.*/,"",u); if(!seen[u]++) n++} END {
  printf "distinct crawled: %d of %d indexable (%.0f%%)\n", n, '"$site_pages"', 100*n/'"$site_pages"'
}' access.log

Expected Output: distinct crawled: 2130 of 5000 indexable (43%). A low and flat coverage ratio on a large site means whole sections go untouched for days — a crawl-budget problem you address by improving internal linking and pruning waste so budget reaches more of the site. Tracking the ratio over weeks, especially after publishing content or fixing waste, is what shows whether your changes actually broadened crawl reach or just shuffled it. Feed the total indexable count from your sitemap or CMS, and the ratio becomes a single KPI for crawl coverage.

Cumulative Coverage Over a Window

A single day undersells coverage because crawlers work through a site over time — a page not crawled today may be crawled tomorrow. Cumulative distinct-URL coverage across a multi-day window is the truer measure of how much of your site search engines eventually reach, and it rises toward your page count as the window lengthens if crawling is healthy.

# Cumulative distinct URLs across a week of logs — true reachable coverage
cat access-2026-06-1[4-9].log access-2026-06-20.log \
  | awk '/Googlebot/ {u=$7; sub(/\?.*/,"",u); seen[u]=1} END {print length(seen)" distinct URLs in 7 days"}'

Expected Output: a distinct-URL total across the week that is higher than any single day — e.g. 3980 distinct URLs in 7 days versus ~2100 per day. If the seven-day cumulative still falls far short of your indexable count, a real chunk of the site is going uncrawled even given time, which points to discoverability problems (weak internal linking, orphan pages) rather than daily budget limits. Comparing the one-day figure to the seven-day cumulative separates "crawlers just haven't gotten to it yet" from "crawlers can't find it at all," which lead to very different fixes.

Watching Coverage After a Site Change

The distinct-URL metric earns its keep around changes: a migration, a new internal-linking scheme, or a sitemap overhaul should move coverage, and the daily count is how you confirm it did. Capture a baseline before the change, then watch the distinct-URL trend after.

# Compare pre/post-change distinct-URL coverage to prove a change worked
pre=$(awk '/Googlebot/ {u=$7;sub(/\?.*/,"",u);s[u]=1} END{print length(s)}' pre_change.log)
post=$(awk '/Googlebot/ {u=$7;sub(/\?.*/,"",u);s[u]=1} END{print length(s)}' post_change.log)
echo "distinct URLs — before: $pre  after: $post"

Expected Output: the two distinct-URL counts side by side, e.g. before: 2130 after: 2620, showing a real broadening of crawl reach after the change. A rising distinct-URL count after adding internal links to previously-orphaned sections is direct evidence the links worked; a flat count means the change did not improve discoverability, whatever else it did. This before/after framing turns a coverage metric into a way to attribute crawl gains to specific work, the same one-variable-at-a-time discipline the broader crawl-budget guides rely on.

Why Distinct-URL Coverage Is the Metric That Matters

Of all the numbers you can pull from a crawl log, distinct-URL coverage is among the most strategically important and the most overlooked, because it answers the question that total hits cannot: is search engine crawling actually reaching your content. A high hit count with low distinct coverage describes a crawler stuck re-fetching a small set — often a handful of high-authority pages or, worse, a parameter explosion — while the rest of the site goes untouched. For a large site whose value lies in breadth, that pattern is a quiet crisis: pages that are never crawled cannot be indexed, and pages that are never indexed cannot rank, regardless of their quality.

This is why distinct-URL coverage deserves a place alongside crawl rate and status distribution as a headline crawl-budget metric. It reframes the goal from "keep crawlers busy" to "get crawlers to the whole valuable site," which is the outcome that actually drives organic visibility. When coverage is low relative to your indexable page count, the fixes are structural — stronger internal linking to surface deep pages, pruning the waste that consumes budget before it reaches new content, and cleaning the parameter and faceted URLs that inflate hits without adding coverage. Tracking coverage over time, and treating a low or declining ratio as a signal to investigate discoverability, is what keeps a large site's crawl-budget program focused on the metric that maps to results rather than the vanity number of total activity.

Diagnosing Low Coverage from the Distinct Count

When the distinct-URL count comes in low, the number itself does not tell you why, but combining it with a few adjacent measurements narrows the cause quickly. If total hits are high but distinct URLs are low, the crawler is churning a small set — look for parameter explosion inflating hits, or a small group of pages being re-fetched far more than they change. If both hits and distinct URLs are low, the crawler simply is not visiting much, which points to server-health throttling or a genuine lack of crawl demand rather than a coverage-distribution problem. And if distinct coverage is low only in specific sections, those sections have a discoverability problem — weak internal linking, orphan pages, or a robots.txt rule accidentally suppressing them.

The distinct count is therefore the starting point of a diagnosis, not its conclusion. Pair it with the hit-to-distinct ratio to see whether the problem is churn or absence, with a per-section breakdown to localize where coverage is thin, and with the crawl-rate-by-hour view to check whether server speed is capping crawl at your busy times. Each combination points at a different fix, and reading them together turns "coverage is low" into "coverage is low in the blog section because those pages are orphaned" — a diagnosis specific enough to act on. This is the analytical habit that makes the distinct-URL metric powerful: it is not a standalone number but the entry point to a structured investigation of why parts of your site are not being crawled, and what to change so they are.

Tracking Coverage as a Standing Metric

Computed once, the distinct-URL count is a spot check; tracked daily, it becomes a standing coverage metric whose trend is more informative than any single value. A gentle upward trend as you add content and internal links confirms crawling is keeping pace with your site's growth; a flat or declining trend despite new content is an early warning that discoverability is not improving. Recording the daily distinct count alongside your other crawl metrics — rate, status mix — gives coverage a permanent place in your crawl-health dashboard, where its trajectory tells you whether search engines are reaching more of your site over time or falling behind. Making coverage a metric you watch continuously, rather than a number you occasionally compute, is what lets you catch a coverage problem while it is still small and act before large sections of the site drift out of the crawl.

Common Mistakes

  • Reporting total hits as coverage. Hits count activity; unique URLs count reach. Track the distinct count for coverage.
  • Leaving query strings in. Parameter URLs balloon the unique count with duplicates of canonical pages. Strip ?... for a true coverage figure.

Frequently Asked Questions

What is a good unique-URLs-per-day number?
There is no absolute target — compare it to your site's indexable page count and its own trend. On a large site, a distinct-URL count far below your page count means large areas go uncrawled; a flat trend after adding content means new pages are not being discovered.

Why not just use sort -u | wc -l per day?
That works for one day but requires a separate pass or file per day. The awk version computes every day's distinct count in a single pass over the whole log, which is faster and scales to large multi-day files.

Part of the CLI One-Liners for Quick Audits guide.