Regex for Security Logs

import re. A raw string r"..." keeps backslashes as regex, not as Python escapes — r"\d" is the pattern digit, while a non-raw "\d" is also a digit only because Python cooperated; get in the habit of raw patterns anyway. re.compile(pattern) builds a reusable object. In a file loop you compile above the loop, not inside it. ip_re.search(line) returns a match object or None. Never call .group() on None — the lab already skipped with if m:. group(0) is the whole match. group(1) is the first capturing parentheses. Named groups (?P<ip>...) exist when one pattern grows; this lesson stays at group(1) so you can see the parentheses. findall returns strings (or tuples of groups) for every match in the line; search returns the first. For one IP per sshd line, search is enough.

Anchors and laziness can wait. You need three ideas: (1) \d is a digit, {1,3} is one to three of them, \. is a literal dot because . means “any character.” (2) (?:...) is a non-capturing group so you can repeat a dotted octet without creating extra group numbers. (3) Parentheses you care about are the capture. A teaching IPv4 pattern is r"(d{1,3}(?:.d{1,3}){3})" — four decimal blobs with dots. It will also match 999.999.999.999, which is not a legal IPv4. That is acceptable for a lab log you wrote with 203.0.113.10. Tightening to 0–255 makes a long pattern people paste wrong at 2 a.m. Readable beats clever. If a line has two dotted quads, search takes the first; write the sample so the source address is the one after from if that is the field you want, or use a more specific prefix r"from (\d{1,3}(?:\.\d{1,3}){3})". Specificity from context beats a monster class.

IPs: prefix from plus the dotted quad, because sshd lines in your fiction look like Failed password for root from 203.0.113.10 port 44022. Status codes: a web-access teaching line like GET /login 401 plus r"\b(401|403|500)\b" or r"\s(4\d{2}|5\d{2})\s" — still on lines you invented, not a scrape of a live site. Failed-login patterns: you can keep the substring Failed password as the cheap gate, then regex the username with r"Failed password for (\S+) from". The username capture is for counting in YOUR sample (how often root vs admin). It is not a wordlist for Hydra. If you take group(1) and feed it to a spray, you left the lesson.

Do not extract email addresses, national IDs, or phone numbers from a real dump “because regex can.” This course does not give you a PII harvester. Do not write a pattern for a real customer log format you smuggled from work. Do not use regex to deanonymize: no “this IP plus this username plus a public search equals a human, so I will contact them.” Matches are fields in a file you own. Module 3 already said documentation IPs are labels. Regex does not change that. IPv6, hostnames, and NAT make real attribution a specialist job with legal process — not a student findall.

Overfit means the pattern only works on the exact spacing of the first line you tried, then silently misses the rest — or it is so greedy it captures half the file. Keep 5–10 fixture lines next to the script. When you change the pattern, run it on all fixtures. If a fixture fails, fix the pattern or split cases (sshd versus nginx) instead of adding fourteen optional groups. Comments in the script should say what field you wanted, not a novel. Verbose re.VERBOSE patterns are allowed later; here, one visual line you can read is the bar. Do not copy a 2,000-character “ultimate IPv4 regex” from a random gist. Do not enable re.DOTALL and then wonder why .* ate the whole log.

Write regex_samples.txt and extract_fields.py in cyberlium-lab. Run it. Copy the printed fields into notes. Do not add a live packet capture. Do not point findall at a mailbox export. chmod 600.

mkdir -p "$HOME/cyberlium-lab"
cd "$HOME/cyberlium-lab"

cat > regex_samples.txt << 'TXT'
sshd: Failed password for root from 203.0.113.10 port 44022
sshd: Failed password for admin from 198.51.100.8 port 51234
nginx: GET /login 401 142
nginx: GET /admin 403 88
sshd: Failed password for ubuntu from 192.0.2.55 port 22022
TXT

cat > extract_fields.py << 'PY'
import re

ip_re = re.compile(r"from (\d{1,3}(?:\.\d{1,3}){3})")
status_re = re.compile(r"\s(401|403|500)\s")
user_re = re.compile(r"Failed password for (\S+) from")

with open("regex_samples.txt", encoding="utf-8", errors="ignore") as f:
    for line in f:
        line = line.rstrip("
")
        ip = ip_re.search(line)
        st = status_re.search(line)
        user = user_re.search(line)
        print("LINE:", line)
        if ip:
            print("  ip:", ip.group(1))
        if st:
            print("  status:", st.group(1))
        if user:
            print("  failed_user:", user.group(1))
        print("---")
print("ethics: fields from MY fixtures — not people, not a spray list")
PY

python3 extract_fields.py

NOTES="$HOME/cyberlium-lab/regex-notes.txt"
{
  echo "=== REGEX NOTES (sample lines only) ==="
  echo "compile: re.compile once above the loop"
  echo "groups: group(1) is the capture"
  echo "ipv4_pattern: from + dotted quad (teaching, not 0-255 perfect)"
  echo "do_not_overfit: 5 fixtures in regex_samples.txt"
  echo "ethics: no prod scrape, no deanonymize, no scan, no Hydra on users"
} > "$NOTES"

chmod 600 regex_samples.txt extract_fields.py "$NOTES"

You now compile a pattern, capture a field, and test it on fixtures you own. IPv4, status, failed-user are the same engine. Overfit and people-finding are the failure modes. Next — Secrets, Ethics & Scope — os.environ, .gitignore, written rules of engagement, localhost versus the internet, and the sentence that will not save you in court: “I was just practicing.”