Deduplicating Load-Balancer Duplicate Hits

When both your load balancer and your origin web servers log requests, a single Googlebot fetch appears twice in the centralized stream — and every per-URL and per-section crawl count you compute is inflated, sometimes by exactly 2×. This page detects that double-logging and removes it correctly, using an event identity that survives the merge. It is the accuracy step that makes the counts from multi-server log centralization real.

Diagnosis: Detect the Double-Logging

If two tiers log the same request, you will see paired lines with the same client, time, and path but possibly different node= tags. Count near-identical adjacent records in the time-ordered merged log:

awk '{k=$1 $2 $4 $5} k==prev{dupes++} {prev=k} END{print dupes" likely duplicate line(s)"}' merged.log

Expected Output: a duplicate count. If it approaches half your total lines, you are almost certainly logging each request at both the balancer and the origin.

Concept: Choosing a Stable Event Identity

Deduplication is only as good as the key that defines "the same event." Too loose (path only) and you merge genuinely distinct requests; too tight (include a per-tier field like the node tag or response time) and true duplicates look different and survive. The stable identity for an HTTP request is the tuple that both tiers agree on: real client IP, precise timestamp, method, and request path. Response bytes and internal tags differ between tiers and must be excluded from the key.

Step-by-Step Fix

Step 1: Decide the authoritative tier. Prefer the origin web nodes — they see the true request path and user-agent, whereas some balancers rewrite or truncate them. The cleanest dedup is to simply not centralize the balancer's access log if the origin already covers every request.

Step 2: If you must merge both, dedupe on the identity tuple. Keep the first occurrence of each unique event.

awk '!seen[$1 FS $2 FS $4 FS $5]++' merged.log > merged.dedup.log
echo "removed $(( $(wc -l < merged.log) - $(wc -l < merged.dedup.log) )) duplicate(s)"

Expected Output: the number of duplicates removed, and a merged.dedup.log with one record per real request.

Step 3: Normalize the client IP first if a proxy rewrote it. Behind a balancer, $remote_addr may be the balancer's IP on one tier and the real client on another, breaking the key. Rewrite to the real client from X-Forwarded-For before dedup, the same real-IP recovery as parsing Cloudflare logs for crawl analysis.

Edge-Case Handling

  • Legitimate retries. A real client retrying a failed request produces two genuine events close in time. The precise timestamp in the key keeps them distinct as long as they differ by at least the log's time resolution; do not deduplicate on path alone or you will erase real retries.
  • Balancer health checks. Internal health-check requests can look like duplicates of nothing. Filter them out by their user-agent or source range before analysis; they are not crawler traffic.

Verification

Confirm a known crawler request now appears once, and that the deduped Googlebot count is roughly half the pre-dedup count if double-logging was total:

grep 'Googlebot' merged.log       | grep ' /products/widget ' | wc -l
grep 'Googlebot' merged.dedup.log | grep ' /products/widget ' | wc -l

Expected Output: the second count is lower — ideally the true number of distinct fetches. If the two counts are equal, the duplicates were not on this path or the key missed them; inspect the tuple.

Why Double-Logging Happens in the First Place

Deduplication is a fix for a problem that is worth understanding at its source, because knowing why requests get logged twice tells you whether you can avoid the duplication rather than clean it up. The most common cause is logging at two tiers that both sit in the request path: a load balancer or reverse proxy logs the request as it forwards it, and the origin web server logs the same request as it serves it. Each tier is doing its job correctly — both are configured to log — but the result is that one crawler request produces two log entries, one at each tier, and when you centralize both logs the request appears twice.

Other causes are subtler. An at-least-once log shipper may redeliver events after a restart or network hiccup, producing duplicates in the destination. A misconfigured agent tailing a file across a rotation may re-read lines it already shipped. A retry at the HTTP layer — a client resending a request the server actually received — produces two genuine requests that look nearly identical. Distinguishing these matters, because some are true duplicates to remove (the same request logged twice) while others are distinct events to keep (a real retry is two separate fetches). The tier-based double-logging is the classic case and the one deduplication most directly addresses, but recognizing that duplication has several possible sources is what keeps you from either missing a duplication cause or wrongly collapsing genuine repeat requests. The best fix, where you can apply it, is to avoid the duplication rather than dedup it — but that is not always possible, which is why the deduplication technique matters.

Choosing the Deduplication Key Correctly

The entire correctness of deduplication rests on the key that defines "the same event," and getting it wrong fails in one of two opposite directions. A key that is too loose — say, the path alone — collapses genuinely distinct requests into one, erasing real repeat crawls and undercounting. A key that is too tight — including a field that differs between the two log entries of the same request — makes true duplicates look distinct, so they survive the dedup and the over-counting persists. The correct key is the tuple of fields that both tiers agree on for the same request and that differs between genuinely distinct requests: client IP, precise timestamp, method, and path.

Each element of that tuple earns its place. The client IP and path identify who requested what; the method distinguishes a GET from a POST to the same URL; and the precise timestamp is what keeps two genuine fetches of the same URL at different moments distinct while collapsing two log entries of the same fetch. The field that must be excluded is anything tier-specific — a load-balancer node tag, a response-time measurement, an internal request ID that one tier adds and the other does not — because those differ between the two entries of the same request and would defeat the dedup. Choosing the key is therefore a matter of identifying exactly the fields that are identical across every log of one real request and different across separate requests, and using precisely those. This is the crux of correct deduplication, and it is why the technique is more subtle than "remove duplicate lines": the lines are not identical, so you must define identity by the fields that matter and ignore the ones that do not.

Verifying the Deduplication Worked

A deduplication pass that runs without error has not necessarily done the right thing, and verifying its effect is what separates a dedup you can trust from one that silently over- or under-collapsed. The verification has two halves, mirroring the two ways the key can be wrong. First, confirm that known duplicates were removed: pick a crawler request you know was double-logged and check that it now appears once, and confirm the total duplicate count removed is roughly what you expected given how many tiers were logging. Second, confirm that genuine distinct requests survived: check that a URL legitimately crawled multiple times still shows its multiple fetches, so you know the key did not collapse real repeat crawls.

The reconciliation check ties it together at the aggregate level. The deduplicated total should equal the sum of the per-tier totals minus the number of duplicates you intended to remove — if it is lower, the key was too loose and erased real events; if it is barely reduced, the key was too tight and duplicates survived. Running this reconciliation on every dedup pass turns "the dedup ran" into "the dedup produced the right count," which is the only assurance that the crawl numbers built on the deduplicated data are accurate. Because deduplication directly sets the denominator of every per-URL and per-section crawl metric, an unverified dedup is a silent risk to every number downstream, and the reconciliation is cheap insurance against it. Making verification an integral part of the dedup, rather than trusting that removing duplicates worked, is what makes the centralized, deduplicated dataset genuinely trustworthy.

Dedup as a Guardian of Count Accuracy

Deduplication's role in the centralized log pipeline is specific and important: it is the guardian of count accuracy, the step that ensures each real request is counted exactly once, and understanding it that way clarifies why it deserves careful attention. Every per-URL, per-section, and per-status crawl metric is a count, and a count is only correct if each real event contributes exactly one to it. Double-logging violates that by making some events contribute twice, and deduplication restores it by collapsing the duplicates — so dedup is precisely the mechanism that keeps the counts, and therefore every metric built on them, accurate.

This guardian role is why a dedup error is so consequential. If dedup removes too much, real events are undercounted; if it removes too little, duplicates inflate the counts; either way, every downstream metric inherits the error. Because dedup sits at the exact point where count accuracy is established, it is the single step whose correctness determines whether the fleet's crawl numbers are right, which is why it is worth the careful key selection and the verification this page describes. Seeing deduplication as the guardian of count accuracy — the step that makes each real request count once and only once — is what gives it the weight it deserves in the pipeline, because the accuracy of every crawl metric a centralized dataset produces depends on the dedup being correct.

Common Mistakes

  • Deduping on path only. This erases legitimate repeat crawls and retries. Include the precise timestamp and client IP in the key.
  • Including per-tier fields in the key. A node= tag or response-time field differs between the two logs of the same request, so true duplicates survive. Key only on fields both tiers share.

Frequently Asked Questions

Should I just log at one tier instead of deduplicating?
Usually, yes. If the origin web nodes already log every request with the true client IP and path, centralize from them alone and skip the balancer's access log entirely — no dedup needed. Deduplication is the fallback when you cannot avoid capturing both tiers.

How do I keep real repeat crawls while removing duplicates?
Use a key that includes the precise timestamp, so two genuinely separate fetches of the same URL — at different instants — stay distinct, while two logs of the same instant collapse to one. Deduping on the URL alone would wrongly merge real repeat crawls.

Part of the Multi-Server Log Centralization for SEO guide.