P10 Parse logs with regex

FreeLesson 62 of 78

P10Parse logs with regexDEHARD

Sounds like: "from this log, report errors per job", "which step is slowest", "parse these lines into records"

Worked question: Parse app.log into records, skip anything malformed, then report per job: lines, errors, slowest step and the most common error.

Approach: 1. write one regex with named groups 2. match line by line, keep skipped lines 3. aggregate with dicts and Counter

STEP 1: LINES TO RECORDSPYTHON 3.12

import re

LINE = re.compile(
    r"(?P<ts>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2})\s+"
    r"(?P<level>INFO|WARN|ERROR)\s+"
    r"job=(?P<job>\w+)\s+step=(?P<step>\w+)"
    r"(?:\s+rows=(?P<rows>\d+))?\s+duration_ms=(?P<ms>\d+)"
    r'(?:\s+msg="(?P<msg>[^"]*)")?'
)

parsed, skipped = [], []
with open("app.log") as f:
    for n, line in enumerate(f, start=1):
        m = LINE.match(line)
        if m:
            parsed.append(m.groupdict())
        else:
            skipped.append((n, line.strip()))

print(len(parsed), "parsed; skipped:", skipped)
print(parsed[4])

Output

13 parsed; skipped: [(7, '--- scheduler restarted ---')]
{'ts': '2025-04-01 02:05:02', 'level': 'ERROR', 'job': 'customers_dim', 'step': 'load', 'rows': None, 'ms': '5003', 'msg': 'timeout connecting to warehouse'}

STEP 2: AGGREGATEPYTHON 3.12

from collections import Counter, defaultdict
# parsed comes from step 1

report = defaultdict(lambda: {"lines": 0, "errors": 0,
                              "slowest": ("", 0)})
for p in parsed:
    r = report[p["job"]]
    r["lines"] += 1
    r["errors"] += p["level"] == "ERROR"
    ms = int(p["ms"])
    if ms > r["slowest"][1]:
        r["slowest"] = (p["step"], ms)

print(f"{'job':<14}{'lines':>6}{'errs':>5}  slowest")
for job, r in report.items():
    step, ms = r["slowest"]
    print(f"{job:<14}{r['lines']:>6}{r['errors']:>5}  {step} {ms}ms")

errors = Counter(p["msg"] for p in parsed if p["level"] == "ERROR")
print(errors.most_common(1))

Output

job            lines errs  slowest
orders_daily       6    0  load 1530ms
customers_dim      7    3  load 5003ms
[('timeout connecting to warehouse', 3)]

Regex piece	Matches
`\d{4}-\d{2}-\d{2}`	A date like 2025-04-01
`\s+`	One or more spaces, so INFO and ERROR line up either way
`(?P<name>...)`	A named group, read back with m.group("name") or groupdict()
`(?:...)?`	An optional, non-capturing chunk
`[^"]*`	Anything up to the next double quote

Practice before you move on

Solve the practice question for P10 in Coding Practice, under Python.
Extend the regex to read rows as a number and total the rows per job.

In Coding Practice

Solve this pattern on fresh data, under Python. Hidden tests grade it.

Parse pipeline logs into a per-job report