Converting Combined Logs to JSON with jq

Modern log tooling — Loki, Vector, DuckDB, jq itself — is happiest with structured JSON, but the logs you already have are legacy combined format. Rather than re-provision every server to emit JSON going forward, you can convert existing combined-format logs to JSON on the command line, one object per request, and feed them straight into the newer stack. This page is that conversion, bridging the combined format to the structured JSON logging approach and its companion parsing JSON access logs with jq.

Diagnosis: The Format Mismatch

Point a JSON-expecting tool at a combined log and it rejects every line:

head -1 access.log | jq .

Expected Output (the symptom): jq: error ... Invalid numeric literal — the combined line is not JSON, so jq cannot parse it. Conversion has to happen first.

Concept: Parse Once, Emit Structured

The reliable conversion parses each line with a capture regex that understands the combined format's quotes and brackets, then assembles the captured groups into a JSON object. jq can do both halves: its capture filter applies a named-group regex, and object construction emits clean JSON. Keeping the request method, path, and query distinct at this stage is what makes the JSON useful for the crawl analysis that follows, per the field discipline in field interpretation and decoding.

Step-by-Step Fix

Step 1: Convert with jq's capture filter. Feed raw lines as strings (-R) and emit one JSON object per line.

jq -R -c '
  capture("^(?<ip>\\S+) \\S+ \\S+ \\[(?<ts>[^\\]]+)\\] \"(?<method>\\S+) (?<path>\\S+)[^\"]*\" (?<status>\\d{3}) (?<bytes>\\d+|-) \"(?<referer>[^\"]*)\" \"(?<ua>[^\"]*)\"")
  | .status |= tonumber
  | .bytes |= (if . == "-" then 0 else tonumber end)
' access.log > access.json
head -1 access.json

Expected Output:

{"ip":"66.249.66.1","ts":"19/Jun/2026:08:30:00 +0000","method":"GET","path":"/shoes?c=red","status":200,"bytes":8421,"referer":"-","ua":"Mozilla/5.0 (compatible; Googlebot/2.1; ...)"}

The -c flag emits compact one-object-per-line JSON (JSON Lines), the format Loki, Vector, and DuckDB ingest directly.

Step 2: Split path and query for crawl analysis. Add a derived field so parameter waste is queryable.

jq -c '. + {url_path: (.path | split("?")[0]), query: (.path | split("?")[1] // "")}' access.json \
  | head -1

Expected Output: each object gains url_path (/shoes) and query (c=red), so you can group by canonical path without the query fragmenting it.

Step 3: Pipe straight into analysis. Once it is JSON Lines, jq answers crawl questions directly.

jq -r 'select(.ua | test("Googlebot")) | .status' access.json \
  | sort | uniq -c | sort -rn

Expected Output: Googlebot status distribution — the same report you would build with awk, now on structured data that any JSON tool can also read.

Edge-Case Handling

  • Lines that fail the regex. capture produces no output for a non-matching line, silently dropping it. Count the drop rate (wc -l before vs after) so a format variant does not vanish unnoticed — the same parse-rate discipline as handling malformed log lines in Python.
  • Embedded quotes in the user-agent. A UA containing an escaped quote can break the [^"]* capture. These are rare; route the few failures to a reject file rather than losing them.

Verification

Confirm no lines were lost and the JSON is valid:

echo "in:  $(wc -l < access.log)"
echo "out: $(wc -l < access.json)"
jq -e . access.json > /dev/null && echo "valid JSON Lines"

Expected Output: matching line counts and valid JSON Lines. A shortfall means the regex missed a variant; inspect the unmatched lines and widen the pattern.

Why Convert Rather Than Re-Log

Faced with combined-format logs and JSON-hungry tooling, there are two paths — convert the existing logs to JSON, or reconfigure the servers to emit JSON going forward — and the right answer is usually both, for different reasons. Reconfiguring servers to log JSON natively is the durable fix: it removes the parsing step permanently, so every future log line arrives self-describing and no downstream tool needs a regex to read it. But that only helps going forward; the history you already have, and any server you cannot reconfigure, remains in combined format. Converting is what brings that existing data into the JSON world.

The conversion is therefore not a workaround to avoid fixing logging properly, but a complement to it. You switch new logging to JSON to eliminate future parsing, and you convert the historical combined logs you still need to analyze so they too can flow into JSON-native tools. There is also a category of logs you cannot control — a third-party service, a legacy appliance, a managed platform that emits combined format — where conversion is the only option, since you cannot change how they log. In all these cases, a reliable combined-to-JSON conversion is the bridge between the format you have and the format your tooling wants. Treating conversion as a permanent capability rather than a one-time migration, applied wherever combined-format logs meet JSON tooling, is what lets you standardize on JSON analysis without needing every log source to cooperate.

Preserving Field Types Through Conversion

A conversion that produces valid JSON but wrong types quietly undermines everything downstream, so getting the types right is as important as getting the structure right. The combined format is entirely text, but its fields have natural types — status and byte count are numbers, the timestamp is a datetime — and a naive conversion that leaves them all as strings produces JSON that sorts and compares incorrectly. A status of "404" as a string sorts lexically, not numerically, and a byte count as a string cannot be summed without coercion, so the conversion must cast each field to its proper type as it builds the object.

The discipline is to coerce numeric fields to numbers and parse the timestamp to a real datetime during conversion, not to defer it to every consumer. A status field emitted as the number 404, a byte count as an integer, and a timestamp as an ISO-8601 datetime string mean every downstream tool receives correctly-typed data and none of them has to re-coerce it. This is where conversion adds value beyond mere reformatting: done well, it does not just wrap the combined fields in JSON syntax but produces a properly-typed structured record, cleaner than the text it came from. Handling the dash that combined format uses for an empty byte count — coercing it to zero or null rather than letting it break the numeric cast — is part of this discipline, since real logs always contain those edge values. Converting with correct types is what makes the resulting JSON genuinely more useful than the combined format, rather than just differently-shaped.

Feeding Converted JSON Into Modern Tooling

The payoff of conversion is that the JSON output slots directly into the modern log stack that combined format cannot enter without a parse step, and knowing where it goes shapes how you convert. JSON Lines — one JSON object per line — is the format that Loki, Vector, DuckDB, and most log tooling ingest natively, so emitting compact one-object-per-line output makes the converted logs immediately consumable. From there, jq can query the JSON directly for a quick analysis, a columnar engine can load it for heavier work, and a log-aggregation system can ingest it without the Grok patterns combined format would require.

This is the reason conversion is worth doing rather than just parsing combined format in place each time: once converted, the data is reusable across every JSON-native tool without re-parsing. A combined log must be re-parsed by every tool that reads it, each applying its own regex and each vulnerable to the same field-position fragility; a JSON log is parsed once, at conversion, and every tool thereafter reads named fields reliably. The conversion is thus a one-time cost that pays off across every subsequent analysis, turning fragile, repeatedly-parsed text into robust, self-describing records that flow through the modern log ecosystem without friction. Where the same JSON feeds many analyses — a jq filter here, a DuckDB log analytics query there, an ingestion into a dashboard — the convert-once economy is exactly what makes standardizing on JSON worthwhile.

Conversion as a One-Time Bridge

The combined-to-JSON conversion is best understood as a one-time bridge between the format you have and the format your tooling wants, and framing it that way clarifies both when to do it and how much effort it deserves. For the historical logs you need to analyze but cannot re-log, conversion is the bridge that brings them into the JSON world so modern tooling can read them; you convert them once, and thereafter they are JSON. For logs from sources you cannot control — a third-party service, a legacy system — conversion is a standing bridge you run continuously, since those sources will keep emitting combined format.

Seeing conversion as a bridge rather than a permanent workflow points to the better long-term move where you have the choice: reconfigure the source to emit JSON natively, and the bridge is no longer needed for that source. So conversion and native JSON logging are complementary — you switch sources you control to JSON, removing the need to convert them, and you convert the sources you cannot control or the history you already have. The bridge framing keeps you from treating conversion as the permanent solution when native logging is available, while still valuing it for the cases where it is the only option. Applying conversion as a deliberate bridge — one-time for history, standing for uncontrollable sources, and unnecessary where you can log JSON natively — is what fits it correctly into a strategy of standardizing on JSON, using it exactly where it is needed and no more.

Convert Once, Benefit Everywhere

The economics of conversion favor doing it once and reaping the benefit across every subsequent analysis. A combined log must be re-parsed by every tool that reads it, each applying its own fragile regex; a converted JSON log is parsed once, at conversion, and every tool thereafter reads named fields reliably. So the one-time conversion cost is repaid by every analysis that no longer has to parse, and the fragility of repeated regex parsing is eliminated across all of them. This convert-once economy is what makes the effort worthwhile: pay a bounded cost at conversion, and every downstream tool reads clean, typed, self-describing records without the parsing step and its risks. For logs that feed many analyses, converting once is unambiguously cheaper than parsing repeatedly.

Common Mistakes

  • Forgetting -R. Without raw input, jq tries to parse each line as JSON and fails immediately. -R reads lines as strings for the regex.
  • Leaving numbers as strings. A status of "200" sorts and compares as text. Coerce numeric fields with tonumber so downstream math works.

Frequently Asked Questions

Should I convert old logs or just switch new logging to JSON?
Do both. Switch your servers to emit JSON going forward (it removes the parsing step permanently), and convert the historical combined logs you still need to analyze. Conversion is a one-time cost per archive; native JSON logging is the durable fix.

Is jq fast enough for large logs?
For moderate files, yes, and it streams line by line so memory stays flat. For very large or repeated workloads, convert once to JSON Lines and load into a columnar engine like DuckDB log analytics, which queries the structured data far faster than re-parsing text each time.

Part of the Structured JSON Logging guide.