Deduplicating Log Events in Vector

When a request is logged by more than one tier — a load balancer and the origin, or an agent that retries delivery — the same event reaches your store twice, and every per-URL crawl count is inflated. Vector's dedupe transform removes duplicates in the pipeline, before they land, so downstream analysis is correct by construction. This page configures it for access logs, extending the Vector.dev pipeline configuration guide and applying the identity thinking from deduplicating load-balancer duplicate hits at ingest time.

Diagnosis: Duplicates in the Stream

If two tiers feed Vector, or an at-least-once source redelivers, your sink shows paired events. A quick check against a known crawler request:

vector tap parse_logs 2>/dev/null | grep '/products/widget' | head

Expected Output (the symptom): two events with identical client IP, timestamp, method, and path but perhaps a different pipeline tag — the same request counted twice.

Concept: A Bounded Recent-Event Cache

The dedupe transform keeps a fixed-size cache of recently-seen field combinations and drops any event whose combination is already in the cache. It is a recent-window dedupe, not a global one — it catches duplicates that arrive close together (the load-balancer-plus-origin case), which is exactly the pattern that matters, while staying bounded in memory. The design decision is which fields define "the same event," and, as always, that key must exclude per-tier fields.

Step-by-Step Fix

Step 1: Add the dedupe transform on the identity fields. Match only on fields both copies share.

[transforms.dedupe]
type = "dedupe"
inputs = ["parse_logs"]
fields.match = ["client_ip", "timestamp", "method", "path"]

[transforms.dedupe.cache]
num_events = 100000

Expected Output: events whose client_ip + timestamp + method + path was seen within the last 100,000 events are dropped; the first copy passes through. Route your sink's input from dedupe instead of parse_logs.

Step 2: Size the cache to your duplicate window. The cache must be large enough that both copies of a request fall inside it. If the balancer and origin logs interleave with thousands of other events between the two copies, raise num_events accordingly.

[transforms.dedupe.cache]
num_events = 500000

Expected Output: a larger recent-window that still catches duplicates separated by heavier interleaving, at the cost of more memory (each cached entry holds only the matched fields, so it stays modest).

Production Warning: Do not include a per-tier or per-delivery field (a pipeline tag, an ingest timestamp, a response-time value) in fields.match. If the two copies differ in that field, dedupe sees them as distinct and both survive — silently defeating the transform. Match only on fields that are identical across every copy of the same request.

Edge-Case Handling

  • Real repeat crawls. A crawler legitimately fetching the same URL at two different instants has two different timestamp values, so the identity key keeps them distinct — dedupe removes double-logging, not real repeat crawls, as long as the timestamp is in the match set.
  • Coarse timestamps. If your timestamp has only second resolution, two genuinely distinct sub-second requests could collide. This is rare for crawler traffic; if it matters, include a higher-resolution field or a request id in the match set.

Verification

Confirm the transform is dropping events and that a known duplicate now appears once:

# Vector exposes component metrics; check the dedupe transform's dropped count
curl -s http://127.0.0.1:8686/metrics | grep -i 'dedupe.*events'

Expected Output: a non-zero count of dropped events from the dedupe component, and a single event for /products/widget in the sink. A zero drop count with visible duplicates means the match fields include a per-tier field — remove it.

In-Pipeline Versus After-the-Fact Deduplication

There are two moments at which you can remove duplicate log events — as they flow through the pipeline, or later in the stored data — and Vector's dedupe transform occupies the first, which has distinct advantages worth understanding. Deduplicating in the pipeline means duplicates never reach your store at all, so you never pay to index, store, or query them, and every downstream consumer sees clean data without needing its own dedup logic. The event is removed once, at the point it flows through, and the correctness propagates everywhere automatically.

After-the-fact deduplication — a DISTINCT in a query, or a periodic cleanup job — has the opposite trade-offs. It handles duplicates the pipeline missed, and it can dedupe across arbitrarily long time windows that an in-pipeline cache cannot span, but it means the duplicates are stored and paid for, and every query that forgets to dedupe sees inflated counts. The two approaches are complementary rather than competing: the Vector dedupe transform handles the common case of duplicates that arrive close together — the load-balancer-plus-origin double-log, the short-window redelivery — cheaply and universally at ingest, while a query-time dedupe on a stable key catches anything that slips past the pipeline's recent-window cache. For crawl data, doing the bulk of deduplication in the pipeline is the efficient default, because it keeps the stored data clean and spares every downstream query from having to remember to dedupe. Reserving after-the-fact dedup for the long-window edge cases the pipeline cannot catch is what completes the coverage without paying to store duplicates you could have dropped at ingest.

Sizing the Recent-Event Cache

The Vector dedupe transform is a recent-window mechanism — it holds a bounded cache of recently-seen event identities and drops any event matching one in the cache — so the cache size directly determines which duplicates it can catch. A duplicate is only caught if both copies fall within the cache window, so the cache must be large enough to span the gap between the two copies of a duplicated event. For load-balancer-plus-origin double-logging, that gap is however many other events interleave between the balancer's log of a request and the origin's log of the same request, which depends on your traffic volume and how the two streams merge.

Sizing the cache is therefore a matter of estimating that interleave and setting the cache comfortably larger. If a few thousand events typically separate the two copies of a request, a cache of tens of thousands catches them with margin; if the streams are more separated, the cache must be larger. The cost of a larger cache is memory, but each cached entry holds only the matched identity fields — not the whole event — so the footprint stays modest even for a large cache. The failure mode of an undersized cache is silent: duplicates separated by more events than the cache holds escape deduplication and inflate your counts, with no error to signal it. This is why erring toward a larger cache is the safe choice for crawl data, where an undetected duplicate corrupts a metric. Understanding that the cache size sets the deduplication window — and matching it to your actual event interleave with margin — is what makes the dedupe transform reliably catch the duplicates it is meant to, rather than silently missing the ones that fall outside too small a window.

Guarding Against Over-Deduplication

Deduplication's dangerous failure is not missing a duplicate but removing a genuine event, and the dedupe transform's match fields are where that risk lives. If the fields you match on do not uniquely distinguish separate real events, the transform collapses distinct requests into one — a real repeat crawl of the same URL, or two different clients that happen to share the matched fields, silently disappearing. For crawl data this means undercounting, which is as wrong as the over-counting deduplication was meant to fix, just in the opposite direction and harder to notice because there is no duplicate to point at.

The guard is to include enough in the match key to distinguish genuinely separate events, while excluding the tier-specific fields that would defeat deduplication of true duplicates. The precise timestamp is the critical element: two log entries of the same request share a timestamp, while two separate fetches of the same URL occur at different moments, so including the timestamp keeps real repeat crawls distinct while still collapsing the duplicate logs of one fetch. Matching on client IP, timestamp, method, and path gives that balance — specific enough that separate events stay separate, shared enough that the two logs of one event collapse. The mistake to avoid is a key so loose it merges real events, which turns a deduplication meant to correct over-counting into a source of under-counting. Verifying after configuration that genuine repeat crawls survive — by checking that a URL known to be crawled multiple times still shows its multiple fetches — is what confirms the key distinguishes real events rather than erasing them. Getting this balance right is what makes the dedupe transform improve data accuracy rather than quietly degrade it.

Choosing Where Deduplication Belongs

Deduplication can happen at several points — in the pipeline as events flow, or later in the stored data at query time — and choosing where it belongs is a design decision that affects both cost and correctness. In-pipeline deduplication, as the Vector transform does it, removes duplicates before they are stored, so you never pay to store or query them and every downstream consumer sees clean data automatically. Its limit is the recent-window nature: it catches duplicates that arrive close together but cannot span arbitrarily long gaps. Query-time deduplication catches anything, across any time span, but requires the duplicates to be stored and every query to remember to dedup.

The efficient design usually puts the bulk of deduplication in the pipeline, where it is cheap and universal, and reserves query-time dedup for the long-window edge cases the pipeline's recent window cannot catch. This matches the tool to the duplicate: load-balancer double-logging and short-window redelivery, which produce duplicates close together, are handled cleanly in the pipeline; the rare duplicate separated by a long gap is caught by a query-time safety net. Deciding where deduplication belongs — in-pipeline for the common close-together duplicates, query-time for the long-span stragglers — is what gives you complete coverage without paying to store duplicates you could have dropped at ingest, and it is the design that keeps the crawl data both clean and economical.

Common Mistakes

  • Matching on too many fields. Including a field that differs between copies makes dedupe a no-op. Match only shared identity fields.
  • A cache smaller than the duplicate gap. If the two copies are separated by more events than the cache holds, the second escapes. Size num_events to the real interleave.

Frequently Asked Questions

Does dedupe guarantee exactly-once delivery?
No — it is a best-effort recent-window dedupe, not a global exactly-once guarantee. It reliably removes duplicates that arrive close together, which covers load-balancer-plus-origin double-logging and short-window source redelivery. For strict exactly-once across long gaps, deduplicate in the query layer on a stable key as well.

Should I dedupe in Vector or just log at one tier?
Logging at a single authoritative tier is the cleaner fix when you control it — no duplicates means no dedupe needed. Use the Vector dedupe transform when you cannot avoid capturing multiple tiers, or when an at-least-once source can redeliver events.

Part of the Vector.dev Pipeline Configuration guide.