Parsing 10GB Logs with Python & pandas Efficiently

Loading a 10 GB access log into pandas with a plain pd.read_csv() reliably ends in a MemoryError, a swap-thrashed host, or a kernel OOM kill — long before you ever count a single bot hit. The fix is not a bigger server; it is a streaming pipeline that holds only a small chunk in memory at a time and folds each chunk into a running aggregate. This guide builds that pipeline for combined-format Apache and Nginx logs so crawl budget analysis runs in well under 2 GB of RAM regardless of file size.

The objective is a chunked, regex-front-ended parser that keeps peak memory constant, casts columns to compact dtypes, and emits a deterministic per-URL bot-traffic count. It builds on the dependency setup in the Python logparser setup cluster and on the same field semantics you use everywhere in log parsing workflows and CLI toolchains.

Diagnosis: MemoryError on the First Read

The symptom is immediate. A naive load of the whole file allocates contiguous RAM for every cell as a Python object, and the process dies:

import pandas as pd
pd.read_csv("access.log", sep=" ", header=None)

Expected Output:

MemoryError

Confirm the file is genuinely large enough to need streaming, and gauge how many lines you are about to fold:

ls -lh access.log && wc -l access.log

Expected Output:

-rw-r--r-- 1 www-data www-data 9.7G Mar 15 02:00 access.log
48213004 access.log

Roughly 48 million lines at 9.7 GB is far past what an in-memory DataFrame of object columns can hold; pandas commonly inflates string-heavy logs 3–5× over their on-disk size, so the effective demand is 30 GB or more. That is the gap the streaming pipeline closes.

Concept: Constant Memory via Chunked Streaming

The core idea is to never materialize the whole dataset. You read a bounded number of rows, reduce them to a small aggregate (a value_counts Series), discard the rows, and repeat — so peak memory is set by the chunk size, not the file size. At the end you concatenate the small per-chunk aggregates and sum them once.

Chunked streaming aggregation keeps memory constant A 10GB log is read in fixed-size chunks; each chunk is parsed, filtered, and reduced to a small per-chunk count, then discarded; the small counts are concatenated and summed into a final result while peak memory stays bounded. 10GB log on disk chunk 1 (50k) chunk 2 (50k) chunk N (50k) parse + filter parse + filter parse + filter small count small count small count concat + sum peak memory = one chunk, constant as N grows

A second, equally important point: the combined log format is not a CSV. The timestamp and request fields are quoted strings containing spaces and brackets, so pd.read_csv(sep=" ") misaligns every field after the first quoted value and produces silently wrong data. The robust approach pre-parses each line with a compiled regex into a structured record, then batches records into DataFrames — getting both correctness and bounded memory. Casting columns to category and Int16 further shrinks each chunk, the same dtype discipline covered in the Python logparser setup guide.

Step-by-Step: Regex Pre-Parse + Chunked Aggregation

Step 1: Compile a combined-format regex and a generator that yields clean records. The generator opens plain or gzip logs transparently and skips lines that do not match, so malformed input never reaches pandas.

import re
import gc
import gzip
import pandas as pd
from typing import Iterator

LOG_RE = re.compile(
    r'(?P<ip>\S+) \S+ \S+ \[(?P<timestamp>[^\]]+)\] '
    r'"(?P<method>\S+) (?P<url>\S+) \S+" '
    r'(?P<status>\d{3}) (?P<bytes_sent>\S+) '
    r'"[^"]*" "(?P<user_agent>[^"]*)"'
)

def iter_parsed(filepath: str) -> Iterator[dict]:
    opener = gzip.open if filepath.endswith('.gz') else open
    with opener(filepath, 'rt', encoding='utf-8', errors='replace') as fh:
        for line in fh:
            m = LOG_RE.match(line)
            if m:
                yield m.groupdict()

Explanation: the generator is lazy — it reads one line at a time, so the file is never loaded whole. Matching with a strict regex is also how you keep going past truncated lines; the dedicated approach to handling malformed log lines in a Python parser extends this with logging and recovery.

Step 2: Batch records into fixed-size chunks and reduce each to a count. Accumulate records until the chunk is full, build a DataFrame, cast to compact dtypes, filter bots, and append only the small value_counts result.

BOT_RE = re.compile(r'bot|crawl|spider|slurp|mediapartners', re.IGNORECASE)
CHUNK_SIZE = 50_000

def process_large_log(filepath: str) -> pd.Series:
    results, chunk = [], []
    for record in iter_parsed(filepath):
        chunk.append(record)
        if len(chunk) < CHUNK_SIZE:
            continue
        results.append(_reduce_chunk(chunk))
        chunk.clear()
    if chunk:
        results.append(_reduce_chunk(chunk))
    if not results:
        return pd.Series(dtype='int64')
    return pd.concat(results).groupby(level=0).sum().sort_values(ascending=False)

Explanation: chunk.clear() drops references to the batch so its memory is reclaimed before the next one fills. Only the tiny per-chunk Series survives between iterations.

Step 3: Define the per-chunk reduction with compact dtypes. Casting status to Int16 and user_agent to category is where the real memory savings land.

def _reduce_chunk(records: list[dict]) -> pd.Series:
    df = pd.DataFrame(records)
    df['status'] = pd.to_numeric(df['status'], errors='coerce').astype('Int16')
    df['user_agent'] = df['user_agent'].astype('category')
    bot_mask = df['user_agent'].astype(str).str.contains(BOT_RE, na=False)
    counts = df.loc[bot_mask, 'url'].value_counts()
    del df
    gc.collect()
    return counts

Explanation: Int16 holds an HTTP status in 2 bytes instead of an 8-byte pointer; category deduplicates the repeated user-agent strings to a small code table. gc.collect() forces reclamation in a long-running process.

Step 4: Run it and read the top wasted-crawl URLs. The result is a sorted Series ready for reporting.

top = process_large_log("access.log")
print(top.head(3))

Expected Output:

/search?q=                  812044
/tag/archive/2019/          203111
/wp-login.php                 98220

These are the URLs bots hit most; high counts on low-value paths are exactly the crawl waste you act on. Normalizing the parsed timestamps first, as in normalizing log timestamp timezones in Python, lets you slice these same counts by hour reliably.

Edge-Case Handling

Gzipped rotated logs. Most history lives in .gz files from rotation. The generator already branches on the extension, so you can fold a whole rotated set without decompressing to disk:

import glob
def process_glob(pattern: str) -> pd.Series:
    parts = [process_large_log(p) for p in sorted(glob.glob(pattern))]
    return pd.concat(parts).groupby(level=0).sum().sort_values(ascending=False)

print(process_glob("access.log*").head(1))

Expected Output:

/search?q=    1904887

Aligning chunked parsing with how files are rotated and retained is covered in the log rotation strategies cluster.

Already-structured JSON logs. If your access logs are JSON rather than combined-format text, skip the regex entirely — a per-line json.loads into the same chunk loop is faster and less brittle. The full treatment is in parsing JSON access logs with jq; the chunked aggregation half of this guide is unchanged.

Production Warning: Run heavy parses against a copy or a read-only mount, and nice/ionice the process. A 48-million-line scan competing for I/O with a live web server can degrade response times even at low CPU. Schedule full-history runs in a maintenance window.

Verification: Cross-Check the Counts

Confirm the streaming result matches an independent count, and that peak memory stayed bounded. A shell count of bot lines should be within a small margin of the summed pandas total (the margin is the malformed lines the regex skipped):

grep -ciE 'bot|crawl|spider|slurp|mediapartners' access.log

Expected Output:

1914203

import tracemalloc
tracemalloc.start()
total = process_large_log("access.log").sum()
peak = tracemalloc.get_traced_memory()[1] / 1e9
print(f"bot rows: {total}  peak GB: {peak:.2f}")

Expected Output:

bot rows: 1904887  peak GB: 0.71

A peak well under 1 GB on a 9.7 GB file confirms the pipeline is streaming, not buffering, and the row total lining up with the grep count confirms correctness.

Common Mistakes

  • Calling pd.read_csv(sep=" ") on combined logs. Quoted fields with internal spaces break alignment, so every column after the request is wrong. Pre-parse with a regex into structured records first.
  • Building a list of all records before chunking. Materializing the full parse into one list negates the memory savings entirely. Reduce each chunk and discard it before reading the next.
  • Leaving columns as inferred object dtype. String-heavy logs inflate 3–5× without category/Int16 casts. Cast inside the per-chunk reduction, before any filtering or grouping.

Frequently Asked Questions

How much RAM does this approach actually need for a 10 GB log?
With 50,000-row chunks, category user-agent columns, and Int16 status codes, peak RAM is typically 0.5–1.5 GB regardless of file size, because only one chunk plus the small running aggregates are ever resident. A naive whole-file load on the same data would demand 15–30 GB.

Should I switch to Dask or Polars instead?
Pandas with chunked regex parsing handles 10 GB comfortably, so optimize it first. Polars offers much faster string operations and native lazy evaluation, which pays off if you process several large files daily; Dask helps when you genuinely need multi-core or out-of-core joins. Migrate for a measured bottleneck, not by default.

What chunk size gives the best memory-to-speed trade-off?
50,000 rows balances per-DataFrame overhead against I/O latency on most VPS instances. Raise it to 200,000 on a high-memory server to cut Python loop overhead; drop it to 10,000 on a host with under 2 GB free. Profile with tracemalloc to confirm peak stays inside your budget.

Part of the Python logparser Setup series.