Building a Kibana Crawl Dashboard

Once your access logs are flowing into Elasticsearch through the pipeline from the ELK Stack log ingestion guide, Kibana turns them into a crawl dashboard the whole team can read — status mix, crawl-by-section, and a daily trend, all interactive and filterable. This page builds that dashboard with Lens, Kibana's drag-and-drop visualization builder, and gets the field mapping right so the charts are accurate.

Diagnosis: Data In, No View

If documents are indexed but Kibana shows nothing, there is no data view (index pattern) yet — Kibana cannot chart an index it does not know about:

GET _cat/indices/nginx-*?v

Expected Output: your nginx-* indices with document counts, confirming data is present. If Kibana's Discover is empty despite this, the missing piece is a data view over nginx-*, not missing data.

Concept: Keyword Fields for Aggregation

Kibana aggregations (the terms behind every bar and pie) require keyword fields, not analyzed text. A user-agent or URL mapped only as text cannot be grouped — Elasticsearch tokenizes it. The crawl dashboard depends on http.response.status_code (a number), a url.path keyword, and a user_agent keyword. Getting these mappings right at index time, as the ELK ingestion guide's Grok pipeline does, is what makes the whole dashboard possible; a field mapped wrong is a chart that will not build.

Step-by-Step

Step 1: Create the data view. In Stack Management → Data Views, add a view over nginx-* with the @timestamp field as the time field.

Expected Output: Discover now shows your log documents on a time axis, and the field list exposes status, url.path, and user_agent as aggregatable (keyword/number) fields — the prerequisite for Lens.

Step 2: Build the status-mix visualization. In Lens, choose a bar chart: vertical axis Count, break down by http.response.status_code, filtered to Googlebot.

Chart:   Bar (stacked)
Filter:  user_agent.original : *Googlebot*
Y:       Count of records
Break down by:  http.response.status_code  (top 6)

Expected Output: a stacked bar of Googlebot requests by status over time — the 200-versus-3xx/4xx split that is your crawl-health headline.

Step 3: Add crawl-by-section and daily-trend panels. A top-values table on the first path segment, and a line chart over the date histogram.

Panel 2 (table):  Top values of url.path, metric Count, filtered to Googlebot, top 15
Panel 3 (line):   Count over @timestamp (daily), split by status class (2xx/3xx/4xx)

Expected Output: a ranked list of the most-crawled sections and a daily productive-versus-wasted line — the trend that shows whether a fix moved crawl toward 200s. Filter every panel to verified crawlers, not just the user-agent string, using the enrichment from detecting fake Googlebot traffic.

Edge-Case Handling

  • User-agent mapped as text only. If user_agent.original will not appear as a filter/aggregation field, add a keyword sub-field in the index template and reindex, or use the ECS user_agent fields the Filebeat nginx module produces.
  • Query strings fragmenting paths. Grouping on the full URL scatters counts across parameter variants. Index a url.path keyword without the query, or use a runtime field to strip it, so sections aggregate cleanly.

Verification

Reconcile a Kibana panel against a direct Elasticsearch count for the same window:

GET nginx-*/_count
{ "query": { "bool": { "filter": [
  { "wildcard": { "user_agent.original": "*Googlebot*" } },
  { "range": { "@timestamp": { "gte": "2026-06-19T08:00", "lt": "2026-06-19T09:00" } } }
]}}}

Expected Output: a count matching the dashboard's Googlebot total for 08:00–09:00. A mismatch means a panel filter differs from the query or a field is mis-mapped — align them before trusting the view.

Why Field Mapping Determines Everything

The single factor that decides whether a Kibana crawl dashboard is easy or impossible to build is how the underlying fields are mapped in Elasticsearch, and getting this right at index time is far cheaper than discovering it wrong at dashboard time. Kibana's aggregations — the terms behind every bar, pie, and table — require keyword or numeric fields, because a terms aggregation groups on exact values. A field mapped as analyzed text is tokenized by Elasticsearch into individual words for full-text search, which means you cannot group on it: a user-agent mapped as text cannot become a clean per-crawler breakdown, and a URL mapped as text fragments into path words rather than grouping as whole paths.

This is why the ELK ingestion pipeline's field mapping is the real foundation of the dashboard, not the dashboard configuration itself. The fields a crawl dashboard groups on — status code as a number, URL path as a keyword, user-agent as a keyword — must be mapped correctly when the index is created, because changing a mapping requires reindexing, which is expensive and disruptive after the fact. The practical discipline is to define an index template that maps these fields correctly before any data is indexed, so every crawl document arrives with the aggregatable fields the dashboard needs. When a Lens panel refuses to offer a field for grouping, the cause is almost always a text mapping where a keyword was needed, and the fix lives in the index template, not the dashboard. Understanding that field mapping is the precondition for every aggregation is what saves you from building half a dashboard before discovering the data cannot support the rest.

Choosing the Right Visualization for Each Question

A dashboard is a set of answers, and matching each visualization type to the question it answers is what makes the dashboard readable rather than a grab-bag of charts. Crawl analysis asks a few recurring questions, each with a natural chart form. "How is crawl volume trending" is a time-series question, answered by a line or bar chart over the date histogram. "How is budget distributed across statuses" is a composition question, answered by a stacked bar showing the 200-versus-3xx-versus-4xx split over time. "Which sections absorb the most crawl" is a ranking question, answered by a top-values table or horizontal bar. "What is my productive-crawl ratio right now" is a single-number question, answered by a metric widget.

Choosing the form that fits the question keeps each panel instantly legible. A stacked bar of status over time shows at a glance whether waste is growing; a top-sections table shows immediately where crawl concentrates; a single big-number productive-ratio widget gives the headline health metric no study required. The mistake to avoid is forcing every metric into the same chart type or picking a form that obscures the answer — a pie chart of twenty sections is unreadable where a ranked bar is clear, and a table of daily counts hides a trend a line would show. Lens helps here by suggesting appropriate chart types from the fields you select, but the judgment of which question each panel answers, and therefore which form fits, is what turns a collection of charts into a coherent dashboard. Building each panel around its question, in the form that answers it fastest, is the design discipline that makes a Kibana crawl dashboard genuinely useful.

Filtering the Whole Dashboard to Verified Crawlers

A crawl dashboard that includes spoofed traffic misleads in proportion to how much spoofing your site attracts, so filtering the entire dashboard to verified crawlers is what makes its numbers trustworthy. Every scraper impersonating Googlebot inflates the crawl-rate panel, and if the spoofs happen to hit error pages, they distort the productive-versus-waste ratio too. Because the dashboard's panels all draw from the same index, the filter belongs at the dashboard or data-view level, applied once so every panel inherits it, rather than added to each panel individually where it is easy to forget on one.

The clean approach is to enrich each crawler hit with a verified-or-not signal during ingestion — a reverse-DNS or IP-range check in the pipeline — and then apply a dashboard-wide filter that restricts every panel to the verified traffic. With the verification done upstream and the filter applied once, the crawl-rate timeseries counts real Googlebot, the section breakdown reflects genuine crawler interest, and the productive ratio measures actual crawler behavior. This matters most when a spoofing spike occurs: without the filter, a flood of fake Googlebot traffic appears as a crawl-rate surge that could trigger a wrong conclusion or an alert, when no search engine changed anything. Applying the verified-crawler filter at the dashboard level, backed by upstream verification, is what keeps every panel grounded in what search crawlers actually did — the same verification principle that runs through all crawl analysis, enforced once for the whole dashboard rather than panel by panel.

The Dashboard as a Shared Team Resource

A Kibana crawl dashboard's value multiplies when it becomes a shared team resource rather than one analyst's private view, and building it with that sharing in mind shapes how you design it. A dashboard that the whole team reads — SEO specialists, SREs, product managers — needs to be legible without its author present: each panel self-explanatory, the most important metric prominent, the layout telling a coherent story about crawl health. This is a different design goal than a personal scratchpad, and meeting it is what makes the dashboard a tool the organization uses rather than one person consults.

The shared-resource framing also means the dashboard becomes a common reference point for crawl-health conversations. When everyone looks at the same dashboard, discussions about whether crawling is healthy, where budget is going, and whether a change helped rest on a shared, consistent view rather than each person's separate analysis. That shared view is worth the design effort to make it clear and trustworthy, because it aligns the team around the same crawl reality. Designing the Kibana dashboard as a legible, shared team resource — self-explanatory panels, a coherent crawl-health story, a common reference everyone trusts — is what turns it from a personal analysis tool into an organizational asset that keeps the whole team aligned on crawl health.

Common Mistakes

  • Aggregating on a text field. Terms aggregations need keyword fields; a text-only user-agent or URL will not group. Fix the mapping at index time.
  • Filtering on the user-agent string alone. It counts spoofs. Filter panels on a verified-crawler field for accurate crawl numbers.

Frequently Asked Questions

Should I use Lens or the older aggregation-based visualizations?
Lens is the current, recommended builder — it is faster to compose and suggests appropriate chart types from your fields. The older TSVB and aggregation-based visualizations still work and offer finer control for complex cases, but for a standard crawl dashboard Lens is quicker and easier to maintain.

How do I keep the dashboard fast on large indices?
Filter panels to the crawler traffic and a bounded time range so aggregations scan fewer documents, ensure the aggregated fields are keyword/number (not runtime-computed on every query), and consider an index lifecycle policy so old data rolls to cheaper tiers, as the ELK ingestion guide's ILM setup arranges.

Part of the ELK Stack Log Ingestion guide.