Streaming Gzip Logs in Python

Rotated access logs are almost always gzipped, and the wrong way to process them — decompress to disk, then read — wastes space and time, while reading a whole file into memory blows up on a multi-gigabyte archive. Python's gzip module streams a compressed file line by line, so you can parse an entire rotation set with flat memory and no temporary files. This page is that pattern, a companion to the memory discipline in parsing 10GB logs with Python pandas efficiently and the broader Python logparser setup guide.

Diagnosis: The Memory Trap

Reading a compressed log the naive way loads the entire decompressed content at once:

import gzip
data = gzip.open("access.log-20260619.gz", "rt").read()   # DON'T
lines = data.splitlines()

Expected Output (the symptom): on a log that decompresses to several gigabytes, the process's resident memory balloons to match, and a large enough file triggers a MemoryError. The .read() is the mistake.

Concept: Iterate, Do Not Materialize

A file object is an iterator of lines, and gzip.open in text mode returns one that decompresses on demand. Iterating it holds only the current line plus the decompressor's small buffer in memory, regardless of file size. The rule is: never call .read() or .readlines() on a log; iterate the handle and reduce each line to a small aggregate immediately, the same never-materialize principle behind efficient large-file parsing.

Step-by-Step

Step 1: Stream one gzipped file. Open in text mode and iterate.

import gzip

def stream(path):
    opener = gzip.open if path.endswith(".gz") else open
    with opener(path, "rt", encoding="utf-8", errors="replace") as fh:
        for line in fh:
            yield line

Expected Output: stream() yields lines from either a .gz or plain file, chosen by extension, with bad bytes replaced rather than raising — so one corrupt byte does not abort the run.

Step 2: Process a whole rotation set with flat memory. Glob the rotated files and chain their streams.

import glob, collections, re

CRAWLER = re.compile(r"Googlebot", re.I)
PATH = re.compile(r'"\S+ (\S+) ')
counts = collections.Counter()

for path in sorted(glob.glob("/var/log/nginx/access.log-*.gz")):
    for line in stream(path):
        if CRAWLER.search(line):
            m = PATH.search(line)
            if m:
                counts[m.group(1).split("?")[0]] += 1

for url, n in counts.most_common(10):
    print(f"{n:8d}  {url}")

Expected Output: the top crawled paths across every rotated file, produced while memory stays flat — only the Counter grows, and it is bounded by the number of distinct URLs, not the log size.

Step 3: Confirm memory stays flat. Watch resident memory while it runs.

python3 crawl_count.py & pid=$!; \
while kill -0 $pid 2>/dev/null; do grep VmRSS /proc/$pid/status; sleep 5; done

Expected Output: VmRSS holding roughly constant (tens of megabytes) throughout, not climbing with the file — the proof that streaming works.

Edge-Case Handling

  • Truncated or partial .gz. A rotation caught mid-write can end abruptly; wrap the per-file loop in a try/except (EOFError, gzip.BadGzipFile) so one bad file logs a warning and the run continues, rather than aborting the whole set.
  • Mixed compression in one directory. The extension check routes .gz to gzip.open and everything else to open, so a directory holding both plain and compressed rotations just works.

Verification

Reconcile the streamed line count against zcat | wc -l so you know nothing was skipped:

n = sum(1 for _ in stream("access.log-20260619.gz"))
print("python:", n)

zcat access.log-20260619.gz | wc -l

Expected Output: the two counts match. A shortfall means the error-replacement swallowed a decode issue on some lines — inspect them, but the stream itself did not drop rows.

The Memory Model That Makes Streaming Work

Understanding why streaming keeps memory flat, rather than just accepting that it does, is what lets you apply the pattern confidently to any log size. The key is that a file object in Python is an iterator that produces one line at a time, and gzip.open in text mode wraps the decompressor in exactly such an iterator. When you loop over it, Python holds only the current line plus the gzip decompressor's small internal buffer — a few tens of kilobytes — regardless of whether the file decompresses to a megabyte or a hundred gigabytes. The decompression happens incrementally, lazily, as you pull each line, so at no point does the full decompressed content exist in memory at once.

This is the crucial contrast with the naive approach. Calling .read() or .readlines() forces the entire decompressed content into memory as one object, so a log that expands to several gigabytes demands several gigabytes of RAM and, past a threshold, simply fails. Iterating the handle instead — and reducing each line to a small aggregate immediately, rather than accumulating lines in a list — keeps the memory ceiling constant in the file size. The rule that follows is absolute: never materialize a log. Any operation you can express as "process one line, keep a small running result, discard the line" runs in constant memory over a file of any size, while any operation that collects lines into a growing structure scales its memory with the data and eventually breaks. Internalizing this memory model is what turns "streaming worked on my test file" into "streaming will work on any file," because you understand the property that guarantees it rather than having observed it once.

Parallelizing Across Rotated Files

A single stream is memory-flat but single-threaded, and when you need to process a large rotation set faster, the natural parallelism is across files: each rotated log is independent, so a pool of workers can stream different files concurrently and merge their small per-file aggregates at the end. This scales throughput with cores while keeping each worker memory-flat, because every worker is still streaming its own file one line at a time rather than loading it.

from concurrent.futures import ProcessPoolExecutor
import glob, collections

def count_bots(path):                        # each worker streams one file
    c = collections.Counter()
    for line in stream(path):                # the flat-memory generator
        if "Googlebot" in line:
            c[extract_path(line)] += 1
    return c

total = collections.Counter()
with ProcessPoolExecutor(max_workers=4) as pool:
    for partial in pool.map(count_bots, glob.glob("/var/log/nginx/access.log-*.gz")):
        total.update(partial)                # merge small per-file aggregates

Expected Output: the combined crawl counts across the whole rotation set, produced roughly four times faster on a four-core machine, with total memory bounded by the number of workers rather than the data size — each worker holds only its current line and a small Counter. The pattern works because the aggregation is associative: per-file counts merge correctly into a grand total, so splitting the work across files and combining the partial results gives the same answer as processing them sequentially. This is the standard shape for scaling log processing beyond one core without abandoning the flat-memory property — parallelize across the naturally-independent files, keep each worker streaming, and merge the compact results. It is the local-machine equivalent of the fan-out that distributed systems do across nodes, achieved with nothing more than a process pool and the streaming generator.

When to Stream and When to Load Into a Tool

Streaming gzip in Python is the right tool for a specific shape of problem — a custom, line-by-line transformation or extraction over compressed logs — and knowing its boundaries keeps you from using it where a different tool fits better. Streaming excels when you need Python's full expressiveness per line: a complex classification, a stateful transformation, an extraction that a query language cannot express. For those, reading the compressed log directly and processing each line in Python is both efficient and flexible, and the flat-memory property means file size is not a constraint.

Where streaming stops being the best choice is repeated analytical querying over the same data. If you find yourself streaming the same logs again and again to compute different aggregations, the per-query cost of re-reading and re-parsing the compressed files adds up, and the answer is to parse once into a columnar engine that answers many queries fast. A tool like DuckDB log analytics reads the gzipped log directly too, but materializes it into typed columns that subsequent queries hit instantly, which beats re-streaming for any workload that asks more than one question of the data. The dividing line is reuse: stream in Python for a one-pass custom transformation, and load into an analytical engine when the same data feeds many queries. Recognizing which side of that line a task falls on is what keeps you using streaming for what it is genuinely best at — flexible, memory-bounded, single-pass processing — rather than forcing it into a repeated-query role a database handles better.

Streaming as a Default Habit

The deeper lesson of streaming gzip logs is that streaming should be your default habit for any log processing, not a special technique reserved for files that are obviously too big. A log that fits in memory today may not tomorrow as traffic grows, and a script written to load the whole file works until the day it does not, failing exactly when the logs are largest and the analysis most needed. Writing every log-processing script to stream from the start — iterate the handle, reduce each line immediately, never materialize — means the script works regardless of file size, today and as the data grows.

Adopting streaming as the default costs nothing and removes a whole class of future failure. A streaming script is no harder to write than a loading one; it just iterates instead of reading all at once. But the streaming version scales to any file size while the loading version has a ceiling it will eventually hit. So there is no reason to write the loading version except habit, and breaking that habit — defaulting to streaming for every log script — is what makes your log tooling robust against growth. The gzip-streaming pattern here is one instance of a general principle worth internalizing: process logs as streams by default, and file size stops being a constraint you have to think about.

Common Mistakes

  • Calling .read() on a compressed log. It materializes the whole decompressed file in memory. Iterate the handle instead.
  • Decompressing to a temp file first. It doubles disk use and adds an I/O pass. gzip.open reads the compressed file directly.

Frequently Asked Questions

Is streaming gzip slower than decompressing first?
No — it is usually faster end to end because it avoids writing and re-reading a temporary decompressed file. Decompression happens in a C extension as you iterate, so the CPU cost is the same either way, minus the extra disk I/O the temp-file approach adds.

Can I stream gzip logs in parallel across files?
Yes. Each file's stream is independent, so a process pool can parse different rotated files concurrently, then merge the small per-file aggregates. Keep each worker streaming (never .read()) so total memory stays bounded by the number of workers, not the data size.

Part of the Python Logparser Setup guide.