Configuring logrotate for High-Traffic Sites

High-traffic web servers generate access and error logs fast enough to exhaust disk I/O, saturate storage, and — when rotation is configured naively — silently drop the exact crawler requests your SEO analysis depends on. The failure is rarely the volume itself; it is the rotation strategy. A copytruncate race loses milliseconds of traffic on every cycle, a missing postrotate signal sends the server writing to a renamed file, and synchronous gzip on a multi-gigabyte log stalls request processing at the worst moment. This guide gives you precise logrotate configuration, the trigger logic behind it, and validation commands to keep log capture continuous on a busy site.

You will tune the size-versus-time trigger, eliminate the copytruncate race, and move compression off the critical path. Treat this as the rotation layer beneath your log rotation strategies and log retention policies: rotation decides how files are cut, retention decides how long they live, and both must agree or crawl data goes missing.

Diagnosis: Spotting a Rotation-Induced Gap

The signature symptom is Googlebot and Bingbot requests vanishing from daily reports immediately after the rotation cron fires — and sometimes a latency spike at the same minute. Confirm the timing correlation first. Find when rotation ran, then check for a write gap straddling it:

grep CRON /var/log/syslog | grep logrotate | tail -1
awk '{print $4}' /var/log/nginx/access.log | sort | uniq -c | tail -5

Expected Output:

Jun 19 03:00:01 web01 CRON[8120]: (root) CMD (/usr/sbin/logrotate /etc/logrotate.conf)
     91 [19/Jun/2026:02:59:58
     88 [19/Jun/2026:02:59:59
      2 [19/Jun/2026:03:00:01
     95 [19/Jun/2026:03:00:03
     97 [19/Jun/2026:03:00:04

That collapse from ~90 requests/sec to 2 at 03:00:01 — exactly when logrotate ran — is the gap. Lines written during the rotation window went to the truncated or renamed file and never reached your parser. Compare bot counts immediately before and after the boundary to size the loss:

grep -c -i 'googlebot\|bingbot' /var/log/nginx/access.log.1

Expected Output:

13420

If yesterday's file shows materially fewer bot hits than its neighbors, the rotation strategy is dropping crawler traffic.

Concept: The copytruncate Race Window

logrotate offers two ways to cut a busy log, and the difference is the whole game.

With create, logrotate renames the active file (access.log → access.log.1) and creates a fresh access.log, then a postrotate signal tells the server to reopen its log descriptor. Because the rename is atomic and the server keeps writing to the open inode until it reopens, no line is lost — the brief overlap simply lands in the .1 file.

With copytruncate, logrotate copies the file, then truncates the original in place. The server never reopens; it keeps writing to the same descriptor at the same offset. Any request logged in the gap between the copy finishing and the truncate completing is written past the copy's end and then erased by the truncate. On a high-traffic site that window logs dozens of requests, and they are simply gone. The diagram below shows where the lost writes fall.

copytruncate race window vs create + reopen Two timelines: copytruncate copies then truncates with a gap where concurrent writes are lost, while create renames atomically and the server reopens with no loss. copytruncate: race window loses writes 1. copy log to .1 2. truncate original RACE WINDOW writes here are lost create + postrotate: no loss 1. rename to .1 2. create new log 3. postrotate: reopen overlap lands in .1, nothing lost

Beyond the write mechanism, two triggers decide when rotation fires: time (daily) and size (maxsize/size). On bursty traffic a daily-only trigger lets a log balloon to tens of gigabytes between cycles; maxsize adds a safety rotation when the file crosses a threshold ahead of schedule. The art is pairing a time trigger for predictable cadence with a size ceiling for burst protection, then keeping compression off the synchronous path so neither trigger stalls request handling.

Step-by-Step: A Rotation Config That Does Not Drop Traffic

Step 1: Replace copytruncate with create plus a reload signal.
This is the single change that closes the race window.

# /etc/logrotate.d/nginx
/var/log/nginx/*.log {
    daily
    rotate 14
    missingok
    notifempty
    compress
    delaycompress
    maxsize 500M
    create 0640 www-data adm
    sharedscripts
    postrotate
        [ -f /var/run/nginx.pid ] && kill -USR1 $(cat /var/run/nginx.pid)
    endscript
}

Explanation: create renames atomically and postrotate sends SIGUSR1 so Nginx reopens its descriptor with zero loss. maxsize 500M adds a burst-safety rotation between daily cycles, delaycompress keeps the newest rotated file uncompressed for real-time parsers, and sharedscripts runs the reload once no matter how many files matched.

Step 2: Validate the config syntax before it runs unattended.

sudo logrotate -d /etc/logrotate.d/nginx 2>&1 | tail -6

Expected Output:

considering log /var/log/nginx/access.log
  log needs rotating
rotating log /var/log/nginx/access.log, log->rotateCount is 14
considering log /var/log/nginx/error.log
running postrotate script
... (no syntax errors)

Step 3: Move compression off the request path with zstd.
Synchronous gzip on a multi-gigabyte file pins a CPU and throttles request handling. Zstandard compresses faster at comparable ratios.

    compresscmd /usr/bin/zstd
    compressext .zst
    compressoptions -3 --long
    uncompresscmd /usr/bin/unzstd

Explanation: Dropping these directives into the block above swaps gzip for zstd -3, cutting archival CPU time sharply. Pair with delaycompress so compression always runs on a file the server has already stopped writing.

Production Warning: The next step forces a real rotation and fires postrotate, which reloads the web server. Run it only in a maintenance window and only after Step 2 passed clean. A postrotate block that fails (wrong PID path, missing binary) can leave the server writing to a rotated file or, worse, not reloading at all — validate in staging first.

Step 4: Schedule off-peak via a systemd timer, then force one rotation to confirm.
Decouple rotation from the default cron slot so heavy I/O never lands during peak traffic.

# /etc/systemd/system/logrotate-custom.timer
[Unit]
Description=Run logrotate for high-traffic logs

[Timer]
OnCalendar=*-*-* 03:00:00
AccuracySec=1m
Persistent=true

[Install]
WantedBy=timers.target

sudo systemctl daemon-reload
sudo systemctl enable --now logrotate-custom.timer
sudo logrotate -f /etc/logrotate.d/nginx
sudo tail -n 1 /var/log/nginx/error.log

Expected Output:

2026/06/19 03:00:01 [notice] 8123#8123: reopened "/var/log/nginx/access.log"

The reopened line confirms Nginx switched to the fresh file — the proof that create + postrotate worked and no descriptor mismatch remains.

Edge Cases

Apache instead of Nginx. Apache has no SIGUSR1 reopen; use a graceful reload in postrotate, guarded so multiple vhost logs do not trigger repeated reloads:

/var/log/apache2/*.log {
    weekly
    rotate 52
    compress
    delaycompress
    missingok
    notifempty
    create 640 root adm
    sharedscripts
    postrotate
        if invoke-rc.d apache2 status > /dev/null 2>&1; then
            invoke-rc.d apache2 reload > /dev/null
        fi
    endscript
}

sharedscripts ensures the graceful reload fires once per run, not once per matched file.

Inode exhaustion on small-block filesystems. A burst that triggers many maxsize rotations can fill the inode table with uncompressed .1 files even while block usage looks fine. If df -i climbs toward 100% while df -h does not, lower rotate, enable compression sooner, or move archives to a volume with more inodes. This is where rotation hands off to log storage and archival best practices for the cold tier.

Verification

After a real rotation, prove there is no write gap across the boundary by counting requests per second around the rotation minute — the rate should stay flat, not collapse:

grep '19/Jun/2026:03:00:0' /var/log/nginx/access.log /var/log/nginx/access.log.1 \
  | awk -F: '{print $2":"$3":"$4}' | cut -c1-8 | sort | uniq -c

Expected Output:

     93 03:00:00
     95 03:00:01
     96 03:00:02
     94 03:00:03

A steady ~95 requests/sec straight through 03:00:01 — including lines that landed in .1 during the overlap — confirms the rotation lost nothing. Contrast this with the diagnosis output, where the same second held only 2 requests.

Common Mistakes

  • Using copytruncate on a high-concurrency server. It guarantees a race window where in-flight writes are copied then truncated away, corrupting crawl-budget metrics every cycle. Use create plus a postrotate reload so the overlap lands harmlessly in the rotated file.
  • daily with no maxsize. Bursty traffic balloons the active log between cycles, blowing past disk and making the eventual compression a synchronous stall. Always pair a time trigger with a size ceiling for burst protection.
  • Omitting sharedscripts. Without it, the reload command runs once per matched log file, firing several graceful reloads in a row and dropping connections. Wrap the postrotate so the signal fires exactly once per run.

Frequently Asked Questions

Should I use copytruncate or create for high-traffic Nginx servers?
Always use create with a postrotate reload signal. copytruncate opens a race window between the copy and the truncate where in-flight requests are written and then erased, losing milliseconds of traffic every cycle and corrupting crawl-budget metrics. create renames the file atomically and the SIGUSR1 reload makes Nginx reopen its descriptor, so the brief overlap lands in the rotated file and nothing is lost.

How do I keep logrotate from blocking request handling during peak hours?
Move compression off the synchronous path and schedule rotation off-peak. Use delaycompress so compression runs on an already-closed file, swap gzip for zstd -3 to cut CPU time, add maxsize 500M so a burst rotates before the file is unmanageable, and trigger the run with a systemd timer set to a verified low-traffic window rather than the default cron slot.

Why are my log analysis pipelines missing bot traffic right after rotation?
That is a file-descriptor mismatch: the server is still writing to the renamed .1 file because postrotate never signaled it to reopen. Confirm by checking the error log for a reopened line after rotation. Enforce create with correct permissions and a working postrotate block, and verify the PID path and reload command actually run in staging before relying on them.

Part of the Log Rotation Strategies series.