Building a Crawl-Budget Dashboard in Datadog

Ad-hoc log queries answer a question once; a dashboard answers it continuously. If your access logs already flow into Datadog, a few facets and log-based metrics turn them into a live crawl-budget view — status mix, crawl rate, and top crawled sections, refreshing on their own. This page builds that dashboard, extending the ingestion covered in the CloudWatch and Datadog log integration guide.

Diagnosis: Logs Ingested, But Not Queryable by Crawl

If logs are arriving but you cannot slice them by crawler, the fields are not yet facets. In the Log Explorer, a search like @http.useragent:*Googlebot* returns nothing structured:

service:web source:nginx @http.useragent:*Googlebot*

Expected Output (the gap): results appear as raw text but you cannot group or measure by user-agent or status — the fields must be parsed and promoted to facets before any dashboard widget can aggregate them.

Concept: Facets, Measures, and Log-Based Metrics

Datadog aggregates on facets (indexed attributes) and measures (numeric facets you can do math on). A crawl dashboard needs the user-agent and status as facets and, ideally, response size as a measure. Because indexed-log retention is finite and metered, the durable pattern is a log-based metric: a count (or distribution) generated from matching logs at ingest and retained cheaply as a time series, so your crawl-rate graph survives long after the raw logs age out. This is the same "aggregate at ingest, retain the summary" economy as partitioning in BigQuery SEO log analysis.

Step-by-Step

Step 1: Parse and facet the key fields. Ensure a log pipeline extracts http.useragent, http.status_code, and http.url_details.path, then create facets on them from the Log Explorer's field menu.

Expected Output: the user-agent, status, and path become groupable facets. Confirm by grouping a query by @http.status_code and seeing a count per status.

Step 2: Create a log-based metric for verified crawler hits. Define a count metric from the query, tagged by status.

Query:   source:nginx @http.useragent:*Googlebot*
Metric:  seo.googlebot.hits   (count)
Group by (tags): @http.status_code, @http.url_details.path

Expected Output: a metric seo.googlebot.hits that increments per Googlebot request, tagged by status and path — the time series every widget will chart. Filter to verified crawlers where possible; a user-agent match alone counts spoofs, so pair it with the verification thinking from detecting fake Googlebot traffic.

Step 3: Build the dashboard widgets. Add a timeseries and a top-list.

Widget 1 (timeseries):  sum:seo.googlebot.hits{*} by {http.status_code}
Widget 2 (top list):    sum:seo.googlebot.hits{*} by {http.url_details.path}, top 15
Widget 3 (query value): sum:seo.googlebot.hits{http.status_code:200} / sum:seo.googlebot.hits{*}

Expected Output: a stacked crawl-rate-by-status chart, a top-crawled-sections list, and a single "productive crawl ratio" number — the 200-to-total share that is your headline crawl-health metric.

Edge-Case Handling

  • Sampled or excluded logs. If an exclusion filter drops a share of logs before indexing, facet-based counts undercount. Base crawl metrics on log-based metrics generated before exclusion filters, which see every matching log.
  • User-agent match includes spoofs. The metric counts anything claiming Googlebot. For a clean number, add a verified-crawler tag upstream (from a reverse-DNS enrichment) and filter the metric on it.

Verification

Cross-check the dashboard's Googlebot count for an hour against a raw log count for the same window:

grep '19/Jun/2026:08' access.log | grep -c Googlebot

Expected Output: a number close to the dashboard's sum:seo.googlebot.hits for that hour. A large gap points to an exclusion filter or a facet parse issue — reconcile before trusting the trend.

Designing the Dashboard Around Decisions

A crawl-budget dashboard is only useful if its widgets map to decisions, so the design principle is to build each panel around a question someone will act on rather than a metric that happens to be available. The core questions a crawl dashboard should answer at a glance are: is crawl volume healthy and stable, is budget going to productive pages or waste, which sections absorb the most crawl, and is anything trending the wrong way. Each of those maps to a widget — a crawl-rate timeseries, a productive-versus-waste ratio, a top-sections list, and a trend comparison — and each should be readable in seconds, because a dashboard that requires study is a dashboard no one checks.

This decision-first framing also tells you what to leave off. A dashboard cluttered with every available metric buries the few that drive action, so resist the urge to add panels just because the data exists. The productive-crawl ratio — the share of verified crawler hits returning 200 — is the single most decision-relevant number, because it directly measures whether budget is being spent well, and it deserves prominence. The status-distribution and top-sections panels support it by showing where any waste is concentrated. Keeping the dashboard focused on the handful of metrics that actually change what you do, arranged so the most important is the most prominent, is what makes it a tool people use rather than a wall of graphs they ignore. A good crawl dashboard answers "is crawl healthy, and if not where is the problem" faster than running a query would, which is the entire point of building it.

Log-Based Metrics Versus Live Log Queries

Datadog lets you build crawl widgets two ways — querying the indexed logs directly, or charting a log-based metric generated from matching logs at ingest — and the choice has real consequences for cost and retention. Querying indexed logs is flexible and immediate: you write a log query, and the widget shows results across whatever log-retention window you pay for. But indexed-log retention is finite and metered, so a widget that queries raw logs loses history as logs age out, and heavy dashboard queries against large log volumes get expensive. For a live view over recent data, log queries are fine; for anything you want to keep or watch continuously, they are the wrong foundation.

Log-based metrics solve both problems. A log-based metric extracts a count or distribution from matching logs as they arrive and stores it as a long-retention time series at metric prices, decoupled from raw-log retention entirely. A crawl-rate metric generated at ingest persists for months regardless of how long you keep the underlying logs, and charting it costs a fraction of querying the logs it came from. The pattern that follows is to build the dashboard's standing panels — crawl rate, error rate, the trends you watch over time — on log-based metrics, reserving direct log queries for ad-hoc investigation of recent data. This is the same "aggregate at ingest, retain the summary" economy that governs cost-effective log analysis everywhere: compute the metric once as the data flows in, keep the cheap summary, and let the expensive raw logs age out on their own schedule. Building the dashboard on metrics rather than live queries is what makes it both durable and affordable.

Keeping Crawl Metrics to Verified Traffic

A dashboard is only as trustworthy as the data behind it, and a crawl dashboard built on raw user-agent matching includes every scraper impersonating Googlebot, which inflates the crawl-rate panel and distorts the productive-versus-waste ratio. For the dashboard's numbers to inform real decisions, they must reflect verified crawler traffic, which means the verification has to happen upstream of the metrics — either by enriching logs with a verified-crawler tag before they are indexed, or by filtering the log-based metrics to a verified signal.

The practical approach is to add the verification as a log-pipeline step: a reverse-DNS or IP-range check that tags each crawler hit as verified or unverified before the log is indexed and before the log-based metrics are generated. The dashboard's metrics then key on the verified tag, so the crawl-rate panel counts real Googlebot and the waste ratio reflects genuine crawler behavior rather than spoof-contaminated numbers. Without this, a burst of spoofed traffic claiming to be Googlebot would show up as a crawl-rate spike, potentially triggering an alert or a wrong conclusion about crawl health, when in fact no search engine changed its behavior at all. Building verification into the pipeline that feeds the dashboard — rather than trusting the user-agent — is what keeps every panel grounded in what search crawlers actually did, which is the difference between a dashboard that guides decisions and one that occasionally misleads them. This is the same verification discipline the detecting fake Googlebot traffic guide applies, moved upstream so every metric inherits it.

Keeping the Dashboard Trustworthy Over Time

A dashboard built correctly can still drift into unreliability as the underlying data and pipeline change, so maintaining its trustworthiness is an ongoing task rather than a one-time build. The two things that erode a crawl dashboard are a verification signal that stops working — if the upstream reverse-DNS or range check that tags verified crawlers breaks, the dashboard silently starts counting spoofs again — and a facet or metric that stops populating because a log-format change broke its extraction. Neither announces itself; the dashboard keeps rendering, just with wrong or missing data.

Guarding against this drift means periodically reconciling the dashboard against a known-good source — a direct query of the raw logs for the same window — to confirm the numbers still match, and watching for panels that go flat or empty, which signal a broken extraction. Because the dashboard informs decisions, a silent drift into wrong numbers can mislead those decisions, so the reconciliation is worth doing on a schedule rather than assuming the dashboard stays correct. Treating the dashboard as something to verify periodically, not build once and trust forever, is what keeps it a reliable decision aid as the logs and pipeline evolve around it.

The Dashboard's Job Is to Prompt Action

A crawl-budget dashboard succeeds when it prompts the right action faster than running a query would, and judging every design choice by that standard keeps it useful. The dashboard exists so that anyone can see, at a glance, whether crawl is healthy and where any problem lies, and act on it — not to display every available metric. So the most decision-relevant numbers get prominence, the layout tells a coherent crawl-health story, and clutter that would bury the signal is left off. A dashboard built this way answers "is crawl healthy, and if not where" in seconds, which is exactly the prompt-to-action its purpose demands. Designing it around the actions it should prompt, rather than the data it could show, is what makes it a tool the team uses to decide rather than a wall of graphs they scroll past.

Common Mistakes

  • Charting raw indexed logs for long-term trends. Indexed retention is finite and costly; the crawl-rate trend vanishes when logs age out. Use log-based metrics for anything you want to keep.
  • Counting user-agent matches as verified crawlers. Spoofs inflate the metric. Enrich with verification upstream and filter on it.

Frequently Asked Questions

Why use log-based metrics instead of just querying the logs?
Indexed logs are retained for a limited, metered window, so a query-based widget loses history as logs expire. A log-based metric captures the count as a long-retention time series at ingest, so your crawl-rate trend persists cheaply for months regardless of raw-log retention.

How do I keep the dashboard focused on verified crawlers?
Enrich logs with a verification signal upstream — a reverse-DNS check that tags genuine Googlebot — and filter both the log-based metric and the widgets on that tag. Matching the user-agent alone counts spoofed traffic, which distorts every crawl number.

Part of the CloudWatch & Datadog Log Integration guide.