Enriching Logs with GeoIP in Vector

A raw log line has an IP; a useful crawl dataset has a country and a network. Enriching each event with GeoIP data in Vector — before it ever reaches your store — lets you segment crawl traffic by region and, more usefully for bot work, sanity-check that "Googlebot" hits come from plausible networks rather than a residential proxy in an unexpected country. This page wires a GeoIP enrichment table into a Vector remap, extending the pipeline built in the Vector.dev pipeline configuration guide.

Diagnosis: IP Without Context

Without enrichment, your events carry an opaque address that tells you nothing about origin:

vector top   # or inspect a sample event

Expected Output (the gap): an event like {"client_ip": "185.220.101.42", "ua": "...Googlebot..."} — a claimed Googlebot with no way to see it is coming from an unexpected network. GeoIP adds that context.

Concept: Enrichment Tables in Vector

Vector loads a local database (a MaxMind-format .mmdb GeoIP file) as an enrichment table and lets a VRL transform look up a value by IP with get_enrichment_table_record. The lookup happens in-process at pipeline speed, so every event is annotated without an external service call. This is the same in-line, before-storage transformation approach used to redact PII in GDPR-compliant log anonymization techniques.

Step-by-Step Fix

Step 1: Declare the enrichment table. Point Vector at a GeoIP .mmdb file at the top level of the config.

# vector.toml
[enrichment_tables.geoip]
type = "geoip"
path = "/etc/vector/GeoLite2-City.mmdb"

Expected Output: on start, Vector loads the database; a bad path fails fast at boot with a clear error, so you know immediately if the file is missing.

Step 2: Look up the client IP in a remap transform. Use get_enrichment_table_record keyed on the IP.

[transforms.geo]
type = "remap"
inputs = ["parse_logs"]
source = '''
  row, err = get_enrichment_table_record("geoip", { "ip": .client_ip })
  if err == null {
    .geo_country = row.country_code
    .geo_city = row.city_name
  }
'''

Expected Output: each event gains geo_country (and geo_city) fields; the err guard means an IP the database does not know simply passes through without those fields rather than failing the transform.

Step 3: Use ASN to vet crawler claims. With a GeoLite2-ASN database as a second table, add the autonomous system, then flag Googlebot claims from networks that are not Google.

source = '''
  asn, e = get_enrichment_table_record("geoip_asn", { "ip": .client_ip })
  if e == null { .asn_org = asn.autonomous_system_organization }
  if contains(string!(.ua), "Googlebot") && !contains(string!(.asn_org ?? ""), "Google") {
    .suspect_bot = true
  }
'''

Expected Output: events claiming to be Googlebot from a non-Google network are tagged suspect_bot: true — a fast, cheap first filter that complements the authoritative reverse-DNS check in detecting fake Googlebot traffic.

Production Warning: GeoIP and ASN are heuristics, not proof — Google operates many networks and CDNs sit in front of origins, so suspect_bot is a lead, not a verdict. Never block traffic on the ASN heuristic alone; confirm with forward-confirmed reverse DNS before acting.

Edge-Case Handling

  • CDN in front of origin. .client_ip may be the CDN's address, not the real client, which makes GeoIP describe the edge node. Enrich the real client from X-Forwarded-For instead, the real-IP concern from CDN log analysis for SEO.
  • Stale database. GeoIP data drifts; a monthly database refresh keeps country and ASN mappings accurate. Automate the download and a Vector reload.

Verification

Confirm enrichment is populating by sampling events and counting how many gained a country:

# route a copy to a console sink briefly, or query your store
vector tap parse_logs 2>/dev/null | grep -c geo_country

Expected Output: a non-zero count that tracks total events. A zero count means the lookup key name (ip) does not match the database's expected key, or the .mmdb failed to load — check the boot log.

Why Enrich at Ingest Rather Than Query Time

Geographic and network context can be added to log data at two points — as it flows through the pipeline, or later when you query it — and enriching at ingest, as Vector's enrichment tables do, has advantages that make it the right default for crawl analysis. When enrichment happens in the pipeline, the geographic and ASN fields are computed once and stored with each event, so every downstream query reads them directly without recomputing. The lookup cost is paid a single time, at ingest, rather than repeated on every query that needs the context, and the enriched fields become a permanent, queryable dimension of the stored data.

Query-time enrichment — joining logs against a GeoIP dataset when you run a report — has the opposite cost profile. It keeps the stored data smaller and lets you change the enrichment logic without reprocessing, but it repeats the lookup on every query, and it requires every consumer to know how to do the join, which is error-prone and easy to forget. For crawl analysis, where the geographic and network context of a request is a stable property you will query repeatedly — segmenting crawl by region, flagging crawler claims from unexpected networks — computing it once at ingest and storing it is both cheaper overall and simpler for consumers. The enriched fields are just there, on every event, ready to filter and group on. This is the same "compute the derived value once, at the point of ingestion, and let every reader benefit" principle that governs efficient log pipelines generally, applied to geographic enrichment: do the lookup as the data flows in, and the context becomes a free dimension of every subsequent analysis rather than a cost each query re-pays.

GeoIP as a Bot-Verification Signal

Beyond geographic segmentation, GeoIP and ASN enrichment serve a specific crawl-analysis purpose: a cheap first-pass check on whether a crawler is what it claims to be. A request whose user-agent says Googlebot but whose ASN belongs to a residential ISP or an unrelated hosting provider is suspicious, because genuine Googlebot comes from Google's own networks. Enriching each event with its autonomous system organization at ingest means every crawler hit carries the network context needed to flag these mismatches immediately, without a per-request DNS lookup.

The value of this as a first-pass filter is speed. Forward-confirmed reverse DNS is the authoritative bot-verification method, but it costs a DNS round-trip per unique IP, which is expensive at scale; an ASN check is an in-memory lookup that happens during enrichment and costs nothing per query. So the practical pattern is to use the ASN enrichment to flag suspects cheaply — a Googlebot claim from a non-Google network gets a suspect tag — and reserve the expensive reverse-DNS verification for confirming those suspects before any action. The enrichment does not replace verification; it prioritizes it, turning a flood of crawler claims into a short list worth verifying. This layered approach — cheap ASN flag at ingest, authoritative DNS confirmation for flagged suspects — is what makes bot verification tractable at high volume, and it is only possible because the network context is enriched onto every event as it flows through the pipeline. The GeoIP enrichment thus does double duty: geographic segmentation for analysis, and a fast, free first line of bot-verification that feeds the more expensive authoritative check.

Keeping the GeoIP Database Current

GeoIP data is a snapshot of a constantly-shifting reality — IP allocations change hands, networks are reassigned, and ASN registrations update — so a GeoIP database goes stale, and enrichment against a stale database silently produces wrong geographic and network attributions. An IP that a year-old database maps to one country or network may now belong to another, and because the enrichment happens invisibly at ingest, the staleness does not announce itself; it just quietly degrades the accuracy of every geographic segmentation and every ASN-based bot flag built on the enriched data.

The maintenance discipline is to refresh the GeoIP database on a regular schedule — monthly is a common cadence, matching how often the major GeoIP providers publish updates — and to automate the refresh so it does not depend on someone remembering. The automation downloads the current database, validates it loads correctly, and prompts Vector to reload the enrichment table, so the pipeline always enriches against reasonably current data. The cost of skipping this is subtle but real: a stale database does not break the pipeline, so nothing alerts you, but the geographic reports drift from reality and the ASN-based bot flags start misfiring — flagging legitimate crawler IPs whose network moved, or missing spoofs from networks the old database mislabels. Because the enrichment is only as accurate as the database behind it, keeping that database fresh is what keeps the enriched fields trustworthy, and automating the refresh is what keeps the freshness from depending on memory. A GeoIP enrichment is a living dependency, not a one-time setup, and treating it as such is what keeps the geographic and network context it adds genuinely accurate.

Enrichment as Investment in Every Downstream Query

Enriching logs at ingest is best understood as an investment that every downstream query then draws on for free, and that framing explains why in-pipeline enrichment is worth the setup even though it adds a step to the pipeline. The enrichment computes the geographic and network context once, as the event flows through, and stores it on the event, so every query that later needs that context — every geographic segmentation, every ASN-based bot check — reads it directly without recomputing. The one-time enrichment cost at ingest is repaid by every query that would otherwise have to do the lookup itself.

This investment framing also clarifies when in-pipeline enrichment is worthwhile. If the enriched context is queried repeatedly — as geographic and network context is in crawl analysis — enriching once at ingest is clearly cheaper than repeating the lookup on every query, so the investment pays off. If the context were needed only once, query-time enrichment might suffice. For crawl work, where the geographic and network dimensions are queried again and again, the enrichment is a sound investment that makes every one of those queries simpler and cheaper. Treating in-pipeline enrichment as a one-time investment that every downstream query draws on — rather than an extra pipeline step to justify — is what makes the case for it clear: you pay once at ingest and every subsequent query benefits, which for repeatedly-queried context is unambiguously the efficient choice.

Common Mistakes

  • Enriching the CDN IP. Behind a proxy, GeoIP on $remote_addr describes the edge, not the visitor. Enrich the forwarded client IP.
  • Treating ASN as verification. It is a heuristic. Use it to prioritize, then verify suspects by reverse DNS before any block.

Frequently Asked Questions

Does GeoIP enrichment slow the pipeline down?
Minimally. The lookup is an in-memory query against a memory-mapped database, so it adds microseconds per event and no network round-trip. It is far cheaper than enriching downstream in your query layer, and it means the geo context is stored with the event once rather than recomputed on every query.

Can I verify Googlebot with GeoIP alone?
No. GeoIP and ASN narrow suspicion — a Googlebot claim from a residential ISP is a strong lead — but the authoritative test is forward-confirmed reverse DNS. Use enrichment to flag suspects cheaply at ingest, then verify those before acting.

Part of the Vector.dev Pipeline Configuration guide.