Finding Uncrawled Sitemap URLs from Logs

The single question here: which URLs does my sitemap advertise that search engines have not actually fetched? The answer is a set difference between two lists — sitemap URLs and crawled URLs — and the whole trick is making the two lists comparable before you subtract them. This is the diagnostic core of the parent XML sitemap crawl optimization guide.

Diagnosis: Confirm the Symptom

If large parts of your sitemap are being ignored, Search Console shows "Discovered — currently not indexed," but it does not tell you which specific URLs went unfetched. The log does. A first, rough count of the gap:

# how many sitemap paths never appear in the crawl log
comm -23 <(sort -u sitemap_paths.txt) <(sort -u crawled_paths.txt) | wc -l

Expected Output: a number — say 4120 of 18423 sitemap URLs never fetched. A large fraction is the symptom worth investigating.

Concept: Why the Two Lists Must Match Exactly

comm compares sorted lines byte for byte, so any cosmetic difference — a host prefix, a trailing slash, a query string — registers as a mismatch and inflates the "uncrawled" set with false positives. The real work is normalization: reduce both sides to bare, sorted, de-duplicated paths so a true difference is the only difference that survives.

Step-by-Step Fix

Step 1: Build the sitemap path list. Flatten sitemaps to <loc> URLs, then strip host and query.

curl -s https://example.com/sitemap.xml \
  | xmllint --xpath '//*[local-name()="loc"]/text()' - 2>/dev/null | tr ' ' '\n' \
  | sed -E 's#https?://[^/]+##; s/\?.*//; s#/$##' | sort -u > sitemap_paths.txt

Expected Output: one canonical path per line, e.g. /blog/log-analysis. The trailing-slash strip (s#/$##) prevents /blog and /blog/ from counting as different pages.

Step 2: Build the crawled path list. Take verified Googlebot hits and normalize identically.

awk '/Googlebot/ {print $7}' /var/log/nginx/access.log \
  | sed -E 's/\?.*//; s#/$##' | sort -u > crawled_paths.txt

Expected Output: the distinct set of paths Googlebot fetched, normalized to match the sitemap list.

Step 3: Subtract. Lines in the sitemap but not the crawl log are the uncrawled set.

comm -23 sitemap_paths.txt crawled_paths.txt > uncrawled.txt
wc -l uncrawled.txt; head uncrawled.txt

Expected Output: uncrawled.txt with the never-fetched paths — the list to triage. Verify the bots first with verifying Googlebot with reverse DNS so spoofed hits do not mask a URL as "crawled."

Edge-Case Handling

  • Case sensitivity. URLs are case-sensitive on most origins; do not lowercase during normalization or you will merge distinct paths. If your platform treats them case-insensitively, lowercase both lists consistently.
  • www vs apex, http vs https. Strip the scheme and host (as above) so a canonical-host redirect does not make every URL look uncrawled.

Verification

Sanity-check a handful of "uncrawled" URLs by grepping the raw log for each — a match means your normalization is off, not that the URL was crawled:

while read p; do
  grep -q " ${p}[/? ]" /var/log/nginx/access.log && echo "FALSE POSITIVE: $p"
done < <(head uncrawled.txt)

Expected Output: no FALSE POSITIVE lines. Any that print mean a slash or query difference is leaking — tighten the sed normalization and re-run.

Interpreting What the Uncrawled Set Means

A list of uncrawled sitemap URLs is data, not yet a diagnosis, and the value comes from interpreting why each URL is absent from the crawl. There are three broad reasons a listed URL goes uncrawled, and they call for opposite responses. The first is simply time: on a large site, crawlers work through URLs over weeks, so a URL uncrawled in a short window may just not have been reached yet — which is why judging the uncrawled set requires a window of several weeks, not a single day. The second is low crawl demand: search engines decided the URL is not worth crawling, usually because it is low-value, thin, or duplicative, in which case it may not belong in the sitemap at all. The third is a blocker: a robots.txt rule, a noindex that discourages recrawl, or a lack of internal links that leaves the URL undiscoverable.

Distinguishing these is what turns the list into action. A valuable page that is uncrawled because it is orphaned needs internal links, not removal; a thin parameter URL that is uncrawled because search engines rightly ignore it should be pruned from the sitemap; a URL uncrawled because robots.txt blocks it reveals a contradiction to resolve. The uncrawled set, in other words, is a set of questions — for each URL, why is this not being crawled, and what does that tell me to do — rather than a to-do list of pages to force into the crawl. Reading it that way, with the three causes in mind, is what separates a productive sitemap audit from a naive attempt to make everything crawl.

Sampling Versus Full Comparison at Scale

On a site with hundreds of thousands of URLs, comparing the full sitemap against the full crawl set is still fast with comm, but the follow-up work — investigating each uncrawled URL — does not scale, so a sampling strategy keeps the audit tractable. Rather than trying to resolve every uncrawled URL individually, group the uncrawled set by section or template and investigate representatives, since URLs sharing a template usually share a cause. If a hundred product URLs under one category are all uncrawled, the cause is almost certainly structural — a linking or template problem affecting the whole category — not a hundred separate issues.

This grouping turns an unmanageable list into a handful of patterns. Bucket the uncrawled URLs by their path prefix or by the template that generated them, count each bucket, and rank by size; the largest buckets are the highest-leverage fixes because resolving the structural cause fixes every URL in the bucket at once. A single representative from each large bucket tells you whether the cause is orphaning, a robots rule, or genuine low value, and the fix applies across the group. This pattern-based approach is what makes sitemap coverage analysis practical at scale — you are not auditing URLs, you are auditing the structures that produce them, and the log's uncrawled set is the input that reveals which structures are failing to attract crawl.

Feeding the Findings Back Into the Sitemap

The audit closes the loop only when its findings change the sitemap, and the discipline is to treat the uncrawled analysis as a recurring input to sitemap generation rather than a one-time cleanup. URLs that are consistently uncrawled and low-value should be dropped from the sitemap, tightening it to the pages search engines actually want; URLs that are valuable but uncrawled because of discoverability problems should trigger internal-linking fixes, with the sitemap left intact so it keeps signaling the URL exists. Over successive audits, this feedback steadily improves the sitemap's signal-to-noise ratio.

The measurable outcome is a rising crawl-coverage ratio: as you prune the perpetually-uncrawled dead weight and fix the discoverability of the valuable uncrawled pages, the fraction of listed URLs that actually get crawled climbs toward one. Tracking that ratio across audits is how you know the process is working — a sitemap where ninety-five percent of listed URLs are crawled is sending a clean, trusted signal, while one where half go uncrawled is diluting its own value. Making the uncrawled analysis a standing part of how the sitemap is maintained, rather than an occasional investigation, is what keeps the file honest as the site changes, so it continues to earn the crawl budget it consumes rather than wasting it on URLs search engines have already decided to ignore.

Turning the Uncrawled List Into a Recurring Signal

Run once, the uncrawled-URL diff is a snapshot; run on a schedule, its trend becomes a standing signal of sitemap health, and automating it is what turns a one-time audit into continuous monitoring. Because a site's sitemap and crawl pattern both change constantly, the uncrawled fraction is a moving number worth watching: a rising uncrawled percentage means either the sitemap is accumulating dead weight or crawling is falling behind, and catching that trend early is far more useful than discovering a large uncrawled set at some later manual review.

Automating the diff is straightforward — the same extract, normalize, and comm steps wrapped in a scheduled job that records the uncrawled count each run — and the payoff is a time series you can alert on. When the uncrawled fraction crosses a threshold or trends upward, the alert prompts an investigation while the problem is still small. This recurring signal fits alongside the other standing crawl-health metrics, giving sitemap coverage the same continuous attention that crawl rate and error ratio get. Making the uncrawled-URL analysis a scheduled, trended signal rather than an occasional manual check is what keeps the sitemap honest over time, so it continues to earn the crawl budget it consumes as both the site and the crawl pattern evolve around it.

Distinguishing Discovery Lag from Genuine Neglect

The most important interpretive skill with an uncrawled-URL list is telling discovery lag apart from genuine neglect, because they look identical in a single snapshot but call for opposite responses. Discovery lag is a valuable URL that crawlers simply have not reached yet; it will be crawled given time, and needs nothing but patience. Genuine neglect is a URL crawlers have decided is not worth fetching; it will stay uncrawled, and needs either a discoverability fix or removal from the sitemap. The way to distinguish them is time: a URL that stays uncrawled across a long window, well past the point where crawlers would have reached it, is genuinely neglected, while one that appears in the crawl on a later look was merely lagging. Judging the uncrawled set against a window long enough to rule out lag is what turns the raw list into a reliable diagnosis of which URLs actually need attention.

Common Mistakes

  • Comparing unsorted or un-normalized lists. comm needs sorted input and exact matches; skip either and the result is noise.
  • Using one day of logs. Crawlers reach URLs over weeks. Judge "uncrawled" over a window of at least a month.

Frequently Asked Questions

Why does a URL show as uncrawled when I know it ranks?
Almost always a normalization mismatch — a trailing slash, a query string, or a host prefix on one list but not the other. Run the false-positive check above; if it flags the URL, fix the sed step rather than trusting the diff.

Should I delete uncrawled URLs from the sitemap?
Not automatically. First decide whether each is a valuable page that deserves better internal linking or a low-value URL that should not be in the sitemap at all. Prune the latter; strengthen links to the former, then re-measure.

Part of the XML Sitemap Crawl Optimization from Logs guide.