Vector.dev Pipeline Configuration

Vector.dev ingests, parses, and routes web server access logs as a single high-throughput pipeline, turning raw Nginx or Apache lines into structured telemetry you can trust for crawl-budget optimization. This guide builds that pipeline end to end: a file source tailing access logs, a remap transform parsing with Vector Remap Language (VRL), conditional routing that isolates crawler traffic, and sinks with the buffering and acknowledgement settings that keep crawl events from being silently dropped under load. It sits inside the broader Log Parsing Workflows & CLI Toolchains collection as the streaming counterpart to batch parsers.

The throughline of this guide is delivery under pressure. A log pipeline that works on a quiet afternoon is easy; one that does not lose Googlebot hits during a traffic spike or a downstream outage is the actual goal. You will configure sources and transforms, then spend real attention on buffers, backpressure, and malformed-input handling so the pipeline degrades gracefully instead of dropping the very events your crawl analysis depends on.

Key Implementation Objectives:

  • Stand up a validated multi-file Vector configuration run as a non-root agent
  • Parse Combined-format logs with VRL and classify bot traffic
  • Route crawler and error streams to separate, cost-appropriate sinks
  • Tune buffers and acknowledgements to survive backpressure without data loss

Prerequisites

This guide targets a current, stable Vector release (0.34+). Before you begin, confirm a dedicated vector system user exists, that you have a writable /etc/vector/conf.d directory, and that the source logs are in a known format — the built-in parse_apache_log function handles both the Apache and Nginx combined layouts, but custom log_format directives need a parse_regex fallback. You should also have the Vector API enabled (port 8686) so vector top and vector tap can introspect the running topology.

Component Minimum version Role
Vector 0.34+ Ingest, transform, route, and ship logs
VRL bundled Remap language for parsing and classification
nginx / Apache any Source of access.log
curl any Scrape the internal metrics endpoint for verification

If you are choosing between aggregation stacks before committing, the ELK vs Vector.dev vs CloudWatch comparison weighs Vector against the alternatives for SEO log pipelines, and Vector pairs cleanly in front of either the ELK Stack ingestion pipeline or Grafana Loki for SEO log aggregation as the parsing and routing layer.

Environment Initialization & TOML Architecture

Establish a clean directory structure before deploying the agent. Vector relies on TOML for declarative configuration. Use the --config-dir flag to load all .toml files in a directory, keeping sources, transforms, and sinks in separate files so changes stay reviewable.

The topology you are building is shown below: file sources feed a remap (VRL) transform, a route transform splits the stream, and sinks ship each stream onward. The buffer between the route transform and a slow sink is where backpressure is absorbed — annotated in red.

Vector pipeline topology with buffer and backpressure File sources tail nginx and Apache access logs into a remap transform that parses with VRL and classifies bots. A route transform splits the stream into a crawler stream and an error stream. The crawler stream passes through a disk buffer that absorbs backpressure before an HTTP sink; the error stream goes to an archive sink. When the buffer fills, backpressure propagates upstream rather than dropping events. source: nginx file tail source: apache file tail remap (VRL) parse + classify UTC, is_bot route crawler / error disk buffer backpressure HTTP sink analytics archive sink errors / S3 fills then pushes back

Step 1: Create the configuration layout

mkdir -p /etc/vector/conf.d
touch /etc/vector/conf.d/sources.toml
touch /etc/vector/conf.d/transforms.toml
touch /etc/vector/conf.d/sinks.toml
chown -R vector:vector /etc/vector
chmod 640 /etc/vector/conf.d/*.toml

Expected Output: three empty, vector-owned TOML files at mode 640, ready to receive sources, transforms, and sinks respectively.

Production Warning: Never run Vector with root privileges. Create a dedicated vector system user and restrict configuration file permissions to 640. A pipeline that can read every log on the box should not also be able to write outside its own directories.

Step 2: Validate before starting

Vector rejects malformed TOML and VRL before it processes a single event, so make validation a commit gate.

vector validate --config-dir /etc/vector/conf.d/

Expected Output:

√ Configuration is valid

Migrating legacy batch scripts? Compare streaming latency against a Python logparser setup to justify the architectural shift, and reuse that parser's field map so both paths emit the same schema.

Source Configuration for Web Server Logs

Configure file tailing to capture high-throughput access logs. Set read_from = "beginning" for initial backfills; switch to "end" in steady-state production. Use ignore_older_secs to skip aged rotated archives so a fleet restart does not replay weeks of history.

Step 1: Define the file source

# /etc/vector/conf.d/sources.toml
[sources.web_access]
type = "file"
include = ["/var/log/nginx/*.log"]
read_from = "end"
ignore_older_secs = 86400

Step 2: Parse with the built-in function

The parse_apache_log VRL function accepts a format argument ("combined" or "common"). The ! suffix aborts on parse errors; capturing the error with , err = lets you handle malformed input instead of dropping the event.

# /etc/vector/conf.d/transforms.toml
[transforms.parse_logs]
type = "remap"
inputs = ["web_access"]
source = '''
  parsed, err = parse_apache_log(.message, format: "combined")
  if err != null {
    .malformed = true
    .parse_error = err
  } else {
    . = merge(., parsed)
    .timestamp = to_string!(.timestamp)
    .is_bot = (
      contains(to_string!(.agent), "Googlebot") ||
      contains(to_string!(.agent), "Bingbot")
    )
  }
  .parsed_at = now()
'''

Verification Step: Tap the transform output to confirm fields parse correctly:

vector tap parse_logs --outputs-of parse_logs | head -n 1

Expected Output (JSON):

{"agent":"Googlebot/2.1","method":"GET","path":"/sitemap.xml","status":200,"is_bot":true,"parsed_at":"2024-05-21T10:15:00Z"}

Production Warning: A bare parse_apache_log!(...) with the ! suffix aborts the entire event on any parse failure, causing silent data loss the moment a CDN injects an extra field. Capture the error into a .malformed flag and route those events to an archive instead, as covered in parsing malformed JSON logs with Vector VRL.

Transform Pipeline: Parsing, Classification & Field Mapping

Apply VRL to extract SEO-relevant fields and classify traffic. Normalize timestamps to UTC immediately — raw server logs often carry local time offsets that distort crawl windows. Prefer the maintained parse_apache_log over a hand-rolled regex; it handles IPv6 addresses and quoted request strings that custom patterns routinely miss.

The table below maps each parsed field to its placement and crawl-budget purpose, the same discipline you would apply when promoting fields to labels in a downstream store.

Field VRL source Example Type Crawl-budget use
host .host 66.249.66.1 string Verify Googlebot via reverse DNS
timestamp .timestamp 2024-05-21T10:15:00Z timestamp Crawl-rate windows (UTC)
method .method GET, HEAD string Crawlers favor GET/HEAD
path .path /sitemap.xml string Crawl waste, orphan detection
status .status 200, 404 integer Status triage
size .size 521 integer Bandwidth per crawler
agent .agent Googlebot/2.1 string Bot classification
is_bot derived true boolean Routing key

SEO callout — status is an integer after parsing. parse_apache_log returns status and size as integers, so route conditions like .status >= 400 work directly without casting. Mishandling this — comparing .status as a string — is a common cause of a route that silently matches nothing. The class-by-class meaning of each code is detailed in understanding HTTP status codes in server logs.

SEO callout — classify, do not trust, the user agent. The is_bot flag keys routing, but a user-agent string is trivially spoofed. Use it to route, then verify Googlebot by reverse DNS before counting a hit against crawl budget.

Production Warning: Wrap every VRL parse in error-handling branches. An unhandled fallible operation aborts the event, and under load that becomes a steady leak of dropped crawl data that no sink metric will obviously flag.

Routing Logic & Sink Optimization

Split telemetry streams using conditional routing so you isolate crawler traffic from standard user requests and send only relevant events to expensive analytics sinks. Status codes are integers after parsing, so comparisons are numeric.

Step 1: Route and ship

# /etc/vector/conf.d/transforms.toml (continued)
[transforms.route_crawl]
type = "route"
inputs = ["parse_logs"]

[transforms.route_crawl.route]
crawler = '.is_bot == true'
errors  = '.status >= 400'

# /etc/vector/conf.d/sinks.toml
[sinks.seo_analytics]
type = "http"
inputs = ["route_crawl.crawler"]
uri = "https://analytics.internal/api/logs"
encoding.codec = "json"
batch.max_bytes = 1000000
request.concurrency = 10
acknowledgements.enabled = true

[sinks.seo_analytics.buffer]
type = "disk"
max_size = 268435488
when_full = "block"

Explanation: acknowledgements.enabled = true makes the source wait for end-to-end delivery confirmation before advancing its read position, and the disk buffer with when_full = "block" propagates backpressure to the source rather than dropping events when the HTTP endpoint slows. Together they convert "drop on overload" into "slow down on overload."

Verification Step: Monitor sink throughput and retry counts:

curl -s http://localhost:8686/metrics | grep 'component_sent_events_total.*seo_analytics'

Expected Output:

component_sent_events_total{component_id="seo_analytics",component_type="sink",...} 14502

Production Warning: Choosing when_full = "drop_newest" to keep the pipeline fast will discard crawl events during exactly the spikes you most want to measure. For crawl-budget accuracy, prefer block and size the disk buffer to ride out your longest expected sink outage, as detailed in handling high-volume log backpressure in Vector.

Render structured outputs into dashboards using Node.js & GoAccess Integration for real-time crawl visualization.

Validation, Monitoring & Troubleshooting

Implement systematic verification before scaling. Run syntax checks on every configuration commit and monitor pipeline health continuously. Each named failure mode below has a confirming command and a recovery recipe.

# Validate configuration syntax
vector validate --config-dir /etc/vector/conf.d/

# Live pipeline monitoring
vector top

# Debug a specific transform's output
vector tap parse_logs --outputs-of parse_logs

Expected Output (vector top):

┌─────────────────────────────────────────────────────────────────┐
│ Component     │ Events/s │ CPU % │ Memory │ Errors │
├─────────────────────────────────────────────────────────────────┤
│ web_access    │  12,450  │  2.1% │  45MB  │      0 │
│ parse_logs    │  12,450  │  8.4% │ 112MB  │      0 │
│ route_crawl   │  12,450  │  1.2% │  32MB  │      0 │
│ seo_analytics │   3,120  │  3.5% │  28MB  │      0 │
└─────────────────────────────────────────────────────────────────┘

Failure Mode 1: Buffer filling and throughput stalling.
Symptom: vector top shows the source events/s dropping toward zero while a sink's buffer grows. Confirm by reading the buffer gauge.

curl -s http://localhost:8686/metrics | grep 'buffer_byte_size.*seo_analytics'

Recovery: The downstream sink is the bottleneck. Raise request.concurrency, enlarge batch.max_bytes, or expand the disk buffer; with when_full = "block" the pipeline correctly slows rather than dropping, which is the safe failure mode for crawl data.

Failure Mode 2: Silent parse failures.
Symptom: event counts at seo_analytics lag web_access with no visible errors. Confirm by tapping the malformed branch.

vector tap parse_logs --outputs-of parse_logs | grep '"malformed":true' | head

Recovery: A format change is reaching the parser. Inspect the captured .parse_error, extend the VRL, and route .malformed == true events to an archive sink for replay.

Failure Mode 3: Timezone-skewed crawl windows.
Symptom: crawl-rate charts peak in the wrong hour. Confirm by comparing a raw line's offset to the emitted .timestamp.

Recovery: Ensure parse_apache_log output is normalized to UTC and never reintroduce a local-time string downstream; parse_apache_log already emits a UTC timestamp, so the fix is usually removing a later override.

Failure Mode 4: Route matches nothing.
Symptom: the crawler or errors stream sends zero events despite obvious matching traffic.

Recovery: Almost always a type mismatch — comparing .status as a string ("404") instead of the integer it is after parsing. Drop the quotes in the route condition.

Buffering, Acknowledgements & Delivery Guarantees

A pipeline that silently drops crawl events under load produces confident, wrong numbers — the crawl-rate chart looks fine, but it is missing the exact peak-traffic hours you most wanted to measure. Vector's delivery behavior is configurable per sink, and getting it right is the difference between a log dataset you can trust and one you cannot. Two levers matter: the buffer that absorbs bursts, and the acknowledgement setting that ties a source's progress to the sink actually accepting the data.

# Disk buffer survives restarts; block (don't drop) when the sink is behind
[sinks.elasticsearch.buffer]
type = "disk"
max_size = 1073741824      # 1 GiB on-disk buffer
when_full = "block"

[sinks.elasticsearch]
acknowledgements.enabled = true

Expected Output: under a downstream slowdown, events queue on disk and the source applies backpressure rather than discarding data; on a restart, the disk buffer replays what had not yet been delivered. Setting when_full = "block" is the deliberate choice to preserve completeness over raw throughput — the right trade for an SEO dataset where a dropped hour is a hole in every report. The alternative, drop_newest, trades completeness for never blocking the source, which suits pure telemetry but not crawl accounting.

Production Warning: A memory buffer with when_full = "drop_newest" is the quiet default that loses data exactly when it matters most — during a traffic spike. If your crawl numbers must reconcile with reality, use a disk buffer with block, and size it to the longest downstream outage you need to survive. The backpressure mechanics of this choice are explored in depth in handling high-volume log backpressure in Vector.

Acknowledgements close the loop: with acknowledgements.enabled = true, a source that supports it (a file source tracking read position, a queue source) only advances its checkpoint once the sink confirms the event landed. This is what makes an end-to-end at-least-once guarantee real rather than aspirational — a crash between read and write replays the un-acknowledged events instead of losing them. Pair it with a downstream deduplication step, as in deduplicating log events in Vector, so at-least-once does not become double-counted.

Vector as the Collector Beneath Everything Else

A framing that clarifies where Vector fits is that it is most often not the destination for logs but the collector that feeds whatever destination you choose — and understanding this repositions Vector from a competitor of ELK, Loki, and the warehouses to the layer beneath them. Vector's job is to run on every source, tail the logs, parse and transform them into structured events, and ship them reliably to one or more sinks. Those sinks can be Elasticsearch, Loki, a warehouse, an object store, or several at once; Vector does not care, because its role is the collection and transformation, not the storage and query. This is why "ELK versus Vector" is partly a category error, as the comparison guide notes: Vector is frequently the collector that feeds ELK.

Seeing Vector this way explains its design and its value. Because it is the collection layer, it emphasizes reliability — the buffering, acknowledgements, and backpressure handling that ensure events are not lost between source and sink — and flexibility, the ability to transform events in flight and route them to whatever combination of destinations you need. A single Vector deployment can tail logs on every web node, parse them once, redact personal data for compliance, enrich them with geographic context, and fan the result out to a warehouse for long-term analysis and a search index for real-time querying, all from one pipeline. This collector-and-router role is what makes Vector composable with the rest of the toolchain rather than an alternative to it. Treating Vector as the reliable, transforming collection layer that feeds your chosen stores — rather than as a store itself — is what places it correctly in a log architecture, and it is why investing in a solid Vector configuration pays off regardless of where you ultimately keep and query the logs.

Designing Transforms for Reusability

A Vector pipeline's transforms — the parsing, redaction, enrichment, and classification steps — are where its real value lives, and designing them for reusability rather than as a one-off configuration is what makes the pipeline maintainable as it grows. A pipeline that parses the log, redacts personal data, tags bot traffic, and enriches with geographic context is doing several distinct jobs, and structuring each as a clear, single-purpose transform — rather than one monolithic remap that does everything — makes the pipeline easier to understand, test, and modify. When a new requirement arrives, you add or adjust one focused transform rather than untangling a mass of combined logic.

This modular design also makes the transforms testable in isolation, which matters because a transform that silently does the wrong thing corrupts every event that flows through it. A single-purpose redaction transform can be verified to redact correctly on its own; a single-purpose parse transform can be checked for its parse rate independently; a classification transform can be confirmed to categorize crawlers correctly without the confounding effects of the other steps. Vector's configuration supports this modularity naturally, with each transform named and wired to its inputs, so the pipeline becomes a readable graph of focused steps rather than an opaque blob. Designing the transforms as a clear sequence of single-purpose steps, each testable and each doing one thing, is what keeps a Vector pipeline maintainable as it accumulates the parsing, compliance, enrichment, and routing logic that a production log pipeline needs — turning it from a fragile configuration that no one dares touch into a comprehensible pipeline that can evolve with your requirements.

The Pipeline as a Reliability Contract

A Vector pipeline is ultimately a reliability contract: it promises to get every log event from source to destination without loss, correctly transformed, and honoring that contract is what makes the crawl data built on it trustworthy. The buffering, acknowledgements, backpressure handling, and dead-letter routing that this guide configures are all in service of that promise — each guards against a way events could be lost or corrupted between the log being written and the analysis reading it. A pipeline that honors the contract delivers a complete, clean crawl dataset; one that does not silently drops events exactly when the load is highest.

Treating the pipeline as a reliability contract shapes how you build and operate it. Every configuration choice is judged by whether it upholds the promise of complete, correct delivery: a buffer sized to survive outages, acknowledgements that prevent loss on restart, dead-letter routing that preserves what cannot be parsed. And the contract is verified, not assumed — you confirm delivery is complete, that no events are lost under load, that malformed input is preserved rather than dropped. Building and operating the Vector pipeline as a reliability contract, with every choice upholding and every claim verifying the promise of lossless correct delivery, is what makes it a foundation the crawl analysis can trust, because a pipeline that reliably delivers every event is the precondition for a crawl dataset that is complete and accurate.

Common Mistakes

  • Writing a custom regex instead of using parse_apache_log: Hand-rolled patterns fail on IPv6, quoted user-agents containing ", and CDN-added fields. Root cause: reinventing a maintained parser. Fix: use the built-in function and fall back to parse_regex only for genuinely non-standard formats.
  • Using the ! suffix without error capture: parse_apache_log!(...) aborts the whole event on any failure, silently dropping data under load. Fix: capture , err = and route malformed events to an archive instead of discarding them.
  • Choosing drop_newest buffers for speed: This discards crawl events during the traffic spikes you most need to measure. Fix: use when_full = "block" with a disk buffer sized for your longest sink outage.
  • Ignoring timezone offsets: Reintroducing a local-time string after parsing skews crawl-budget windows. Fix: keep the UTC timestamp parse_apache_log emits and never override it with local time.
  • Comparing status as a string in routes: .status == "404" never matches because parse_apache_log returns an integer. Fix: compare numerically, e.g. .status >= 400.

Frequently Asked Questions

How do I filter out internal crawler traffic in Vector.dev?
Apply a route transform condition matching internal IP ranges with VRL's ip_cidr_contains!("10.0.0.0/8", .host) or specific user-agent strings, and direct those events to a blackhole sink so they never reach analytics or inflate crawl counts.

Can Vector.dev handle high-throughput Nginx logs without dropping crawl events?
Yes, if you configure for it. Use a disk-backed buffer with when_full = "block", enable acknowledgements, and tune request.concurrency. That combination converts overload into upstream backpressure rather than dropped events; the deeper tuning is covered in handling high-volume log backpressure in Vector.

What is the best way to normalize timestamps across distributed web servers?
Let parse_apache_log emit its UTC timestamp and avoid any later local-time override. For custom formats, use parse_timestamp in the remap transform with an explicit input format and force conversion to UTC so every server's events share one time base.

How should I handle JSON logs that are sometimes malformed?
Parse with parse_json capturing the error, flag failures with a .malformed field, and route them to an archive for inspection and replay rather than aborting the event — the full pattern is in parsing malformed JSON logs with Vector VRL.

Part of the Log Parsing Workflows & CLI Toolchains series.