Mapping Redirect Hops with Python

An awk one-liner can list every 3xx response, but it cannot tell you that /a → /b → /c is a three-hop chain, because each hop is a separate log line with no link to the next. Reconstructing the chain needs a graph, and that is where Python earns its place: build a map of source paths to their redirect targets, then walk each source to its final destination, surfacing multi-hop chains and loops. This deepens the awk approach in finding redirect chains in logs with awk within the redirect chain optimization guide.

Diagnosis: Hops Look Healthy One at a Time

Every individual redirect returns a valid 301 or 302, so a per-line view sees nothing wrong:

awk '$9 ~ /^3/ {print $7, $9}' access.log | sort | uniq -c | sort -rn | head

Expected Output: a list of redirecting paths, each looking fine — the chain only appears when you connect a redirect's target to another redirect's source, which requires following the graph.

Concept: Redirects Form a Graph

Each redirect is an edge from a source URL to a target URL. Chains are paths through that graph; loops are cycles. To find them you need the target of each redirect, which the access log alone does not record — the log shows the request and its 3xx status, not the Location header. So the reliable method combines the log (to know which URLs crawlers actually hit and redirect) with a probe of each redirecting URL to capture its target, then walks the resulting graph.

Step-by-Step Fix

Step 1: Collect the redirecting URLs from the log. Extract distinct paths that returned a 3xx to a crawler.

import re, subprocess
srcs = set()
LINE = re.compile(r'"\S+ (\S+) \S+" (3\d\d)')
for line in open("access.log", encoding="utf-8", errors="replace"):
    if "Googlebot" not in line:
        continue
    m = LINE.search(line)
    if m:
        srcs.add(m.group(1).split("?")[0])
print(f"{len(srcs)} redirecting paths")

Expected Output: the count of distinct redirecting paths crawlers hit — the set to resolve into chains.

Step 2: Resolve each source to its immediate target. A single non-following request captures the Location.

import urllib.request
def target_of(url):
    req = urllib.request.Request(url, method="HEAD")
    class NoRedirect(urllib.request.HTTPRedirectHandler):
        def redirect_request(self, *a, **k): return None
    op = urllib.request.build_opener(NoRedirect)
    try:
        op.open(req)
    except urllib.error.HTTPError as e:
        return e.headers.get("Location")
    return None

edges = {s: target_of("https://example.com" + s) for s in srcs}

Expected Output: a source → target map, one hop each. Run it against a copy of your site or with polite rate limiting; this issues one request per redirecting URL.

Step 3: Walk each chain to its end and flag problems. Follow edges until a non-redirect or a repeat (loop).

def resolve(start, edges, limit=10):
    seen, cur, hops = [], start, 0
    while cur in edges and edges[cur] and hops < limit:
        if cur in seen:
            return seen + [cur], "LOOP"
        seen.append(cur); cur = edges[cur]; hops += 1
    return seen + [cur], ("CHAIN" if hops > 1 else "OK")

for s in edges:
    path, verdict = resolve(s, edges)
    if verdict != "OK":
        print(verdict, " -> ".join(path))

Expected Output:

CHAIN /products/old -> /products/new -> /shop/new
LOOP  /a -> /b -> /a

Each CHAIN should be collapsed to a single redirect from the original source straight to the final target; each LOOP is a bug that traps crawlers.

Edge-Case Handling

  • Relative Location headers. Some servers return a relative Location; normalize it against the source URL before treating it as the next node, or the walk breaks.
  • Cross-host redirects. A redirect to another domain ends the chain for your purposes — record the external target and stop, rather than probing off-site.

Verification

After collapsing a chain, re-probe the original source and confirm it now reaches the destination in one hop:

print(target_of("https://example.com/products/old"))

Expected Output: the final destination directly (/shop/new), not the intermediate /products/new — proof the chain was flattened. A rising crawl share of 200s afterward confirms recovered budget.

Why the Graph View Beats the Line View

An access log gives you a line-by-line view of redirects — each 3xx response as an isolated event — but redirect problems are structural, and seeing them requires the graph view that connects those events into chains. A single redirect line tells you a URL returned a 301, but it cannot tell you that the target of that 301 is itself a redirect source, because the two facts live on different log lines with nothing linking them. The chain only becomes visible when you reconstruct the graph of source-to-target edges and walk it, which is the whole reason this analysis needs a program rather than a one-liner.

This structural view is what turns redirect analysis from spotting individual redirects into diagnosing the redirect topology of your site. A graph reveals the two structural pathologies that matter: chains, where a sequence of redirects hops through intermediate URLs before reaching a destination, multiplying the crawl cost of every link into the chain; and loops, where the redirects cycle back on themselves, trapping crawlers. Neither is visible in the line-by-line view — a chain looks like a set of individually-valid redirects, and a loop looks like a set of valid redirects that happen to form a cycle. Building the graph and walking it is what surfaces these topological problems, and it is why redirect optimization at any scale beyond a handful of rules benefits from the programmatic, graph-based approach rather than eyeballing redirect lines. The graph is the right mental model for redirects, and the program is what makes that model computable from your logs.

Handling Redirect Graph Edge Cases

Walking a redirect graph in the real world means handling the edge cases that make production redirects messy, and a robust mapper anticipates them rather than breaking on the first one. The most important guard is the loop guard: a redirect cycle would make a naive graph walk recurse forever, so the walk must track the URLs it has visited and stop when it revisits one, reporting the cycle as a loop rather than hanging. This is not a rare edge case — misconfigured redirects produce loops regularly — so the loop guard is essential, not optional.

Other edge cases shape the graph's correctness. A relative Location header — a redirect target given as a path rather than a full URL — must be resolved against the source URL before it becomes a graph node, or the walk breaks at the first relative redirect. A cross-domain redirect, where the target is on another site, ends the chain for your purposes: you record the external destination and stop, since you cannot follow the redirect topology off your own domain. A redirect that depends on request headers — serving different targets to different user-agents or based on cookies — means the graph you build from one probe may not match what a crawler sees, so probing with a crawler-like user-agent matters. Handling these cases — resolving relative targets, stopping at domain boundaries, guarding against loops, and probing as a crawler would — is what makes the redirect map match the reality crawlers actually navigate, rather than an idealized version that breaks on the messiness of production redirects. A mapper that handles them produces a graph you can trust; one that does not gives you a partial or wrong picture exactly where redirects are most tangled.

Integrating the Map Into Redirect Maintenance

A redirect map produced once is a snapshot; integrated into ongoing maintenance, it becomes the tool that keeps redirects healthy as the site changes, and building the mapping into a repeatable process is what turns a one-time diagnosis into continuous hygiene. Because redirects accumulate — every migration and restructure adds more — new chains and loops form over time, so the redirect map needs to be regenerated periodically to catch them before they cost significant budget. A scheduled run of the mapper against the current redirects, flagging any new chains or loops, is what makes redirect problems visible as they arise rather than discovered long after.

The map also serves the proactive side of redirect maintenance. By revealing the full redirect topology, it lets you spot a target that is about to become a chain — a URL that redirects and is itself the target of other redirects — so you can re-point the upstream redirects before the chain forms. Fed into a deploy-time check, the mapping logic can even fail a build that would introduce a chain or loop, preventing the problem at the point of change. So the mapper is not just a diagnostic to run when redirects seem slow but a maintenance tool to run continuously, catching chains and loops as they form and preventing new ones at deploy time. Integrating the redirect map into the standing maintenance of the site — periodic regeneration plus a deploy-time check — is what keeps the redirect topology clean over time, so the crawl budget redirects consume stays minimal rather than silently growing as the site accumulates the redirects that any evolving site produces.

Feeding the Map Back Into Prevention

The redirect map's greatest value is not just diagnosing existing chains but informing how you prevent new ones, and using the map's insight to shape redirect discipline is what stops the same chains from re-forming. By revealing the full redirect topology, the map shows you where chains tend to form on your site — which URL patterns accumulate redirects, which migrations left tangled hops — and that knowledge feeds directly into the rules you set for adding redirects going forward. A team that has seen its redirect map understands where the risks are and can guard against them at the point of change.

This feedback from diagnosis to prevention is what makes the mapping effort compound. The first map cleans up the chains that exist; the understanding it produces prevents new ones, so the redirect topology stays clean rather than needing repeated cleanups. Feeding the map's insight back into a deploy-time check and a redirect-discipline rule turns a one-time diagnostic into a lasting improvement in how the site handles redirects. The map is thus most valuable not as a report you generate when redirects seem slow, but as the source of understanding that shapes a prevention practice keeping the redirect graph clean for good.

Common Mistakes

  • Inferring chains from the log alone. The access log lacks the redirect target. You must capture the Location, then walk the graph.
  • No loop guard. Following edges without a visited-set hangs on a cycle. Track seen nodes and cap the hop count.

Frequently Asked Questions

Why do I need to probe URLs — isn't the target in the log?
No. A standard access log records the request and the 3xx status code, but not the Location response header that names the redirect's target. Without the target you cannot connect one hop to the next, so a lightweight HEAD probe of each redirecting URL supplies the missing edge.

How many hops is too many?
Any chain longer than one hop wastes budget and adds latency, and search engines may stop following after a handful of hops. Collapse every multi-hop chain to a single redirect from the original URL to the final destination; there is no benefit to intermediate hops.

Part of the Redirect Chain Optimization guide.