Comparing Two Log Days with comm

A daily unique-URL count tells you coverage changed; it does not tell you which URLs. comm does: given the crawled-URL set for two days, it splits them into "only yesterday," "only today," and "both," so you can see exactly what Googlebot newly discovered, what it abandoned, and what stayed constant. This page is that comparison, a natural follow-on to counting unique URLs crawled per day with awk within the CLI one-liners for quick audits collection.

Diagnosis: A Count Change Without Detail

If today's unique-URL count jumped or fell versus yesterday, you know something moved but not what. comm turns the aggregate delta into a concrete list:

wc -l crawled_18.txt crawled_19.txt

Expected Output: two counts that differ — say 2130 and 2384. The 254-URL rise could be newly discovered pages (good) or a parameter explosion (bad); comm decides which.

Concept: Set Operations on Sorted Lists

comm reads two sorted files and prints three columns: lines unique to file 1, lines unique to file 2, and lines in both. Suppressing columns with -23, -13, or -12 gives you set difference and intersection directly, with no scripting. The only requirement is that both inputs are sorted and normalized identically — the same discipline that makes the sitemap diff in finding uncrawled sitemap URLs from logs reliable.

Step-by-Step

Step 1: Build a normalized crawled-URL set per day. Strip query strings and sort.

for d in 18 19; do
  grep "${d}/Jun/2026" access.log \
    | awk '/Googlebot/ {u=$7; sub(/\?.*/,"",u); print u}' \
    | sort -u > "crawled_${d}.txt"
done

Expected Output: two sorted files, one per day, of distinct canonical paths.

Step 2: URLs crawled today but not yesterday (newly reached).

comm -13 crawled_18.txt crawled_19.txt | head

Expected Output: paths that appeared in the crawl for the first time today — ideally your newly published pages, confirming discovery.

Step 3: URLs crawled yesterday but not today (dropped).

comm -23 crawled_18.txt crawled_19.txt | head

Expected Output: paths the crawler stopped fetching. A little churn is normal; a whole section disappearing is a signal to investigate for a broken link, a Disallow, or a server error.

Edge-Case Handling

  • Normalize both sides identically. A trailing slash or query string on one day but not the other creates false "new"/"dropped" entries. Apply the same awk/sed to both.
  • Short windows are noisy. Crawlers naturally vary day to day. Compare week-over-week sets, not just adjacent days, to separate signal from noise.

Verification

Confirm the intersection plus the two differences reconcile with the day counts:

echo "both: $(comm -12 crawled_18.txt crawled_19.txt | wc -l)"
echo "only19: $(comm -13 crawled_18.txt crawled_19.txt | wc -l)"
echo "total19: $(wc -l < crawled_19.txt)"

Expected Output: both + only19 = total19. If they do not add up, the files were not sorted or not de-duplicated — rebuild them with sort -u.

Three-Way Comparisons for Trend Direction

Two days tell you what changed overnight; three (or more) tell you the direction of a trend, which is far more actionable. A URL that appeared yesterday and is gone today might be noise, but a URL crawled on day one, absent day two, and back on day three is a different pattern than one steadily fading. Extending the comm approach to several days turns a snapshot into a trajectory.

# Which URLs are newly crawled and STAY crawled across three days (durable discovery)?
comm -12 \
  <(comm -13 crawled_18.txt crawled_19.txt) \
  <(sort -u crawled_20.txt) > durable_new.txt
wc -l durable_new.txt

Expected Output: URLs that first appeared on day 19 and were still being crawled on day 20 — durable discovery, as opposed to a one-day blip. This distinction matters when you have just published content or changed internal linking: a page crawled once and dropped has not really been "discovered," while one that keeps drawing crawl has. Chaining comm operations lets you ask these multi-day questions without a database, and it scales to a weekly cadence by comparing this week's crawled set against last week's rather than adjacent days, which smooths out the daily noise the parent CLI one-liners for quick audits guide warns about.

Comparing by Status, Not Just Presence

Presence in the crawled set is binary, but a URL can be crawled both days and still have changed in a way that matters — from 200 to 404, or 200 to 301. A richer comparison keys on the path and its status, so you catch a URL that started returning an error between the two days even though it appears in both crawled sets.

# Build path+status sets per day, then find URLs whose status changed
for d in 18 19; do
  awk '/Googlebot/ {u=$7; sub(/\?.*/,"",u); print u, $9}' access_${d}.log \
    | sort -u > ps_${d}.txt
done
# Paths present both days but with a different status
join <(sort ps_18.txt) <(sort ps_19.txt) 2>/dev/null | awk '$2 != $3'

Expected Output: paths whose crawler-seen status changed between the days, e.g. /products/x 200 404 — a page that started 404ing overnight, which pure presence comparison would miss because the URL is in both days' crawled sets. This status-aware diff catches regressions the moment they appear in crawl, turning the two-day comparison from "what did crawlers find" into "what broke," and it pairs naturally with the status interpretation in understanding HTTP status codes in server logs.

Scripting a Daily Delta Report

Run manually, the two-day comparison is a spot check; wrapped in a script and cron'd, it becomes a daily early-warning report that surfaces crawl changes without anyone having to look. The script captures each day's normalized crawled set, diffs against the prior day, and emails or logs the new/dropped/changed URLs.

#!/usr/bin/env bash
today=$(date +%Y%m%d); yest=$(date -d yesterday +%Y%m%d)
mk() { awk '/Googlebot/ {u=$7; sub(/\?.*/,"",u); print u}' "$1" | sort -u; }
mk "/logs/access-$today.log"  > "/state/crawled-$today.txt"
echo "== newly crawled ==";  comm -13 "/state/crawled-$yest.txt" "/state/crawled-$today.txt" | head
echo "== dropped ==";        comm -23 "/state/crawled-$yest.txt" "/state/crawled-$today.txt" | head

Expected Output: a dated report listing the day's newly-crawled and dropped URLs, retained as /state/crawled-*.txt snapshots that also give you history for multi-day comparisons later. Because the snapshots accumulate, the same script that produces today's delta also builds the corpus for week-over-week analysis — a small, dependency-free monitoring pipeline that catches an accidental Disallow or a section falling out of crawl the day after it happens, rather than at the next manual audit.

Why Set Comparison Beats Manual Inspection

It is tempting to eyeball two days of logs and spot the differences, but manual inspection fails at exactly the scale where the differences matter. A site crawled at thousands of distinct URLs a day produces sets far too large to compare by reading, and the human eye is precisely bad at noticing that one URL among thousands stopped appearing. Set comparison with comm is not just faster — it is categorically more reliable, because it computes the exact difference deterministically rather than relying on attention. The URL that quietly dropped out of crawl, the section that fell silent, the parameter explosion that appeared overnight: these are invisible to inspection and obvious to a set diff.

This is why the discipline of reducing each day to a normalized, sorted set and diffing it mechanically is worth building into a routine rather than doing ad hoc. The set operations answer questions inspection cannot: not "does today look roughly like yesterday" but "exactly which URLs entered, left, or changed." That precision is what turns a vague sense that "crawl seems different" into a concrete, actionable list — the specific URLs to investigate. And because the operation is deterministic and scriptable, it scales from a manual spot check to an automated daily report without changing the underlying logic, which is the path from occasional curiosity to continuous monitoring that catches crawl regressions the day they happen.

Comparing Across Longer Intervals for Seasonality

Adjacent-day comparison catches sudden changes, but some crawl patterns only reveal themselves over longer intervals, and the same comm approach extended to weekly or monthly snapshots surfaces them. Crawl behavior has rhythms — weekday versus weekend, month-start versus month-end, seasonal content cycles — and comparing this Tuesday's crawled set against last Tuesday's controls for the weekly rhythm in a way that comparing Tuesday against Monday cannot. A URL set that looks volatile day to day may be perfectly stable week over week, and distinguishing genuine change from normal rhythm requires comparing like intervals.

Building a small archive of periodic snapshots — one crawled-URL set per day, retained — makes these longer comparisons trivial after the fact. With daily snapshots on disk, you can compare any two days, any two weeks, or a day against the same weekday a month earlier, all with the same comm operation. This turns the two-day comparison from a point-in-time check into the foundation of a historical crawl-coverage record, where you can ask not just "what changed since yesterday" but "how has crawl coverage of this section trended over the quarter." The snapshots are tiny — a sorted list of paths compresses to almost nothing — so retaining a long history costs little and gives you the interval flexibility that separates catching a sudden regression from understanding a slow trend. The same set-comparison primitive scales seamlessly from overnight change detection to quarter-long trend analysis.

Keeping the Snapshots That Make Comparison Possible

The two-day comparison only works if you have both days' data in a comparable form, which means the real prerequisite is a habit of capturing a normalized crawled-URL snapshot every day. A snapshot is tiny — a sorted list of distinct paths compresses to almost nothing — so retaining months of them costs little and unlocks comparison across any interval you later care about. The discipline is to generate the snapshot as part of your daily log processing, store it dated, and never delete the recent history, so that when a crawl question arises you already have the before-and-after sets rather than wishing you had captured them. This small, cheap habit is what turns the one-off comparison shown here into a standing capability: the ability to answer, at any time, exactly how crawl coverage changed between any two points, because the evidence was quietly accumulating all along.

Common Mistakes

  • Unsorted input. comm assumes sorted files and produces garbage otherwise. Always sort -u first.
  • Comparing raw paths with query strings. Parameter noise makes every day look full of "new" URLs. Strip ?... before comparing.

Frequently Asked Questions

Can I use comm to measure the effect of a change?
Yes. Capture the crawled-URL set before a change (a new internal-linking scheme, a sitemap update) and after, then use comm -13 to list URLs newly crawled as a result. It is a direct, URL-level before/after that aggregate counts cannot give you.

Why not use diff instead of comm?
diff reports line-level edits and is sensitive to order and context, which is wrong for set comparison. comm is purpose-built for "what is in A, B, or both" on sorted sets, which is exactly the question here.

Part of the CLI One-Liners for Quick Audits guide.