Detecting Noindex Pages Still Crawled

Marking a page noindex keeps it out of the index — it does not stop crawlers from fetching it, and a large set of frequently-recrawled noindex pages quietly burns crawl budget on URLs that can never rank. This page finds them: it cross-references your list of noindex URLs against real crawler hits so you can see which ones still draw crawl, and it explains when to leave them alone versus escalate to a block. It extends the sitemap-hygiene work in XML sitemap crawl optimization to the pages you deliberately excluded.

Diagnosis: Confirm the Symptom in the Log

If faceted filters, internal search, or paginated archives carry a noindex tag but still show heavy crawl, the log makes it visible. Count Googlebot hits on a known noindex path prefix:

awk '/Googlebot/ && $7 ~ /^\/search/ {print $7}' /var/log/nginx/access.log | wc -l

Expected Output: a hit count on /search (a typical noindex area), e.g. 9120. Thousands of crawls on pages that can never be indexed is budget you can reclaim.

Concept: Why Noindex Pages Keep Getting Crawled

noindex is a directive a crawler can only obey after it fetches the page and reads the meta tag or header — so the crawl itself must happen for the directive to work. Google recrawls noindex URLs periodically to confirm the tag is still there. That is expected and healthy at low volume. It becomes waste when internal links or a stale sitemap keep pointing crawlers at large noindex spaces, so they are refetched far more than needed. The fix is upstream: reduce the internal-link and sitemap signals that invite the crawl, and only then consider a Disallow.

Step-by-Step Fix

Step 1: Assemble the noindex URL set. From a crawl of your own site (or your CMS), list URLs served with noindex. A quick check for a single URL:

curl -sI https://example.com/search?q=shoes | grep -i 'x-robots-tag'
curl -s https://example.com/search?q=shoes | grep -io '<meta[^>]*robots[^>]*noindex[^>]*>'

Expected Output: an X-Robots-Tag: noindex header or a <meta name="robots" content="noindex"> tag, confirming the page is noindex. Collect the confirmed prefixes (for example /search, /filter, /tag) into noindex_prefixes.txt.

Step 2: Measure crawl on each noindex area. Sum verified crawler hits per prefix.

while read pre; do
  n=$(awk -v p="$pre" '/Googlebot/ && index($7,p)==1 {c++} END{print c+0}' /var/log/nginx/access.log)
  printf '%8d  %s\n' "$n" "$pre"
done < noindex_prefixes.txt | sort -rn

Expected Output: prefixes ranked by crawl volume, e.g. 9120 /search then 3400 /filter. The top rows are where budget leaks.

Step 3: Cut the internal signals first. Before blocking, remove the invitations: drop internal links to the noindex space (add rel="nofollow" is not enough — remove or consolidate the links), and ensure none of these URLs sit in the sitemap. Re-measure after a crawl cycle; often the volume falls without any block.

Edge-Case Handling

  • noindex, follow vs noindex, nofollow. With follow, crawlers still traverse links on the page, spreading crawl further; consider noindex alone (which Google treats as noindex, follow by default) and prune outbound links on high-volume noindex pages.
  • Disallowed and noindex. If you Disallow a URL in robots.txt, crawlers cannot fetch it to see the noindex, so it can linger in the index with no snippet. Never block a URL you are trying to de-index until it has dropped out — the trap detailed in robots.txt and crawl-rate control.

Verification

After trimming links and sitemap entries, confirm crawl volume on the noindex prefixes trends down over the next two to three weeks:

awk '/Googlebot/ && $7 ~ /^\/search/ {split($4,d,":"); print substr(d[1],2)}' /var/log/nginx/access.log \
  | sort | uniq -c

Expected Output: daily hit counts on /search declining after your changes. A flat or rising line means the internal-link or sitemap signal is still feeding the crawl.

Why Noindex Does Not Stop Crawling

The counterintuitive fact at the heart of this analysis is that marking a page noindex does not stop crawlers from fetching it — and understanding why explains both the crawl you observe and the right way to reduce it. A noindex directive lives in the page's meta tag or HTTP header, which a crawler can only read after it has fetched the page. So the fetch must happen for the directive to be seen and obeyed; noindex controls indexing, not crawling. A crawler fetches the page, reads the noindex, and keeps the page out of the index, but the fetch itself — the crawl-budget cost — already occurred.

Worse, crawlers recrawl noindex pages periodically to confirm the directive is still there, since a page's noindex status can change. This recrawl is expected and healthy at low volume — it is how search engines keep their index accurate — but it means noindex pages have an ongoing crawl cost that never goes to zero. The problem arises when that cost is not the modest reconfirmation crawl but a large, ongoing spend driven by internal links and sitemap entries that keep pointing crawlers at noindex spaces far more often than reconfirmation requires. This is the distinction the analysis targets: not the normal, low-level recrawl of noindex pages, which is fine, but the outsized crawl of noindex spaces driven by signals that keep inviting crawlers back. Understanding that noindex cannot stop the crawl — only reduce the indexing — is what tells you the fix lives upstream, in the internal links and sitemap entries that generate the crawl, rather than in the noindex directive itself, which is doing exactly what it is supposed to.

Reducing Noindex Crawl Without Blocking It

Once you have found noindex pages drawing heavy crawl, the instinct to block them with robots.txt is exactly wrong, and understanding why points to the correct fix. Blocking a noindex page in robots.txt prevents the crawler from fetching it, which prevents the crawler from seeing the noindex directive — so a page you were successfully keeping out of the index can drift back in, indexed with no snippet because the crawler can no longer read the noindex that would exclude it. The block breaks the very mechanism keeping the page out of the index. This is the trap that makes noindex-plus-disallow a classic mistake.

The correct approach reduces the crawl by cutting the signals that drive it, not by blocking the fetch. If internal links are sending crawlers to noindex spaces, remove or consolidate those links so the crawler has fewer paths into the noindex area; if the sitemap lists noindex URLs, remove them, since a sitemap should contain only indexable canonical pages. With fewer links and sitemap entries pointing at the noindex space, crawlers naturally visit it less, and the crawl volume falls toward the modest reconfirmation level that is healthy — all while the noindex directive remains visible to the crawlers that do visit, so the pages stay out of the index. Only after a noindex page has fully dropped from the index, if its residual reconfirmation crawl is still worth eliminating, does a robots.txt block become safe. Reducing noindex crawl by trimming its upstream signals rather than blocking its fetch is what lowers the crawl cost while preserving the de-indexing, avoiding the trap of stranding pages in the index by blocking the directive that was keeping them out.

Making Noindex Crawl Part of Routine Auditing

The crawl of noindex pages is not a one-time cleanup but a recurring consideration, because a site continually adds noindex pages — new faceted filters, new internal search results, new paginated archives — each of which can accumulate the internal links and sitemap entries that drive excess crawl. So the durable practice is to include noindex crawl in routine crawl-budget auditing, checking periodically whether any noindex space has grown into a significant crawl consumer and trimming its upstream signals before it does.

Folding this into routine auditing keeps the problem small. A noindex space caught early, when its crawl is just beginning to grow, needs only a light trim of internal links; one left unaudited until it is consuming substantial budget requires a larger intervention and has already wasted crawl in the meantime. The routine audit is the same measurement this page describes — sum crawler hits per noindex prefix, rank them, and investigate any that have grown large — run on a schedule rather than only when someone notices a problem. Because new noindex pages appear as a normal part of site evolution, the audit needs to recur to catch the new spaces before they become significant, just as the crawl-waste and redirect audits recur for the same reason. Making noindex-page crawl a standing item in the periodic crawl-budget review — not a one-time investigation — is what keeps it from silently growing back after you first address it, and it fits naturally alongside the other recurring audits that keep a site's crawl budget optimized as the site changes.

The Broader Lesson About Directive Interactions

The noindex-still-crawled problem teaches a broader lesson worth carrying to every crawl-control decision: directives interact, and controlling crawl well means understanding how noindex, robots.txt, canonical tags, and internal links combine rather than treating each in isolation. Noindex controls indexing but not crawling; robots.txt controls crawling but blocks the crawler from seeing other directives; canonical consolidates signals but does not stop the fetch; internal links drive crawl regardless of the other directives. Each does one specific thing, and using them well requires knowing what each does and does not do, and how they compose.

The classic mistakes in crawl control almost all come from misunderstanding these interactions — blocking a noindex page and stranding it in the index, expecting canonical to stop crawling, assuming noindex reduces crawl. Getting crawl control right means holding the whole set of directives and their interactions in view: to reduce crawl of a page you want de-indexed, you cut the internal links and sitemap entries that drive the crawl while leaving noindex visible, rather than blocking it. This systems view of how the directives combine is what separates competent crawl control from a series of well-intentioned but conflicting individual settings. Carrying the lesson of the noindex-crawl interaction to every crawl-control decision — always asking how the directives compose, not just what each does alone — is what makes crawl steering reliable rather than a source of the contradictory-signal mistakes that come from treating each directive in isolation.

Common Mistakes

  • Blocking to fix crawl before de-indexing. A Disallow stops the fetch that noindex needs, freezing the page in the index. De-index first, block later.
  • Treating all noindex crawl as waste. Periodic recrawl to confirm the tag is normal. Only outsized, link-driven volume is worth acting on.

Frequently Asked Questions

Is it bad that Google crawls my noindex pages at all?
No — a crawler must fetch a page to read its noindex directive, and periodic recrawl to reconfirm the tag is expected. It only becomes a problem when internal links or a sitemap drive far more crawling of noindex spaces than reconfirmation requires.

Should I add Disallow to stop crawling of noindex pages?
Only after the pages have left the index. Disallowing a noindex URL prevents crawlers from seeing the noindex tag, so the page can persist in the index indefinitely. Reduce internal links and sitemap entries first; block only long-crawled, already-dropped URLs.

Part of the XML Sitemap Crawl Optimization from Logs guide.