YOUR CAPSTONE PROJECT · LESSON 17

Capstone: investigate a suspected phishing incident

Build a repeatable investigation that connects email, authentication events, assets and evidence hashes.

55 min · learn at your pace

Your case brief

A fictional payroll email arrived shortly before several failed logons. Build a tool that records what the supplied evidence supports and what remains uncertain. A suspicious URL, matching attachment hash, or failure threshold is a lead; none alone establishes compromise. Nothing in this exercise contacts a host or opens an attachment.

Write investigate(case_dir="case", output_dir="output") in main.py. It returns the summary dictionary and writes five reports. Put a call under the main guard so Run code generates the reports. The supplied case_tools.py contains the UTC parser, URL inspector, attachment manifest and file inventory functions from earlier lessons; you may import them or implement equivalent behavior yourself.

Validate the case before reporting

Require policy.json, indicators.json, assets.json, auth.jsonl, suspicious.eml and the evidence directory. A missing required input raises FileNotFoundError. Keep output outside the case directory. Invalid configuration raises ValueError before writing any new reports.

Policy requires a nonempty string case_id, aware ISO start and end with start before end, and a positive integer threshold excluding booleans. Indicators require domains and sha256 lists: domain strings must normalize to nonempty values with strip().lower().rstrip("."); hashes must contain exactly 64 hexadecimal characters and compare case-insensitively. Assets are a list of nonempty string host and owner fields and a boolean critical. Normalize host keys with strip().casefold() and reject duplicates.

Collect and correlate events

Read nonblank JSONL lines, counting physical lines from one. Require nonempty string id, timestamp, user, host, and outcome; outcome is failure or success, and timestamps must be aware. Strip IDs and treat them as case-sensitive. Normalize users and hosts with strip().casefold(). Keep the first valid occurrence of each ID; reject later occurrences. Invalid records do not reserve an ID.

Record rejected lines in errors.csv using source auth.jsonl and reason invalid JSON, invalid event, or duplicate id. Keep valid records. Accepted-event count covers all valid unique records; the window-event count covers only [start,end). Group windowed failures by user and emit a repeated-failures finding at or above threshold. Its source IDs are sorted and joined with |.

Parse the email from original bytes. Its Date must be valid and aware or the case fails with ValueError. Its source ID is the stripped Message-ID, falling back to email. Only investigate its contents and add its timeline row when its date lies in the window. Use the plain-text body, ignoring HTML. Extract HTTP(S) URL tokens up to whitespace, angle brackets or quotes, trimming trailing ., ,, ;, or ). Inspect their actual hosts with the URL policy from the earlier lesson. An invalid token adds an error with source suspicious.eml, line 0, reason invalid URL.

Emit one email-domain finding per distinct exact matching domain. Subdomains and misleading suffixes do not match automatically. Emit an attachment-hash finding for each matching attachment's decoded hash, using its filename as the subject. Never extract attachments to their claimed paths. These email findings use the message source ID.

Five reviewable reports

  • findings.csv: rule,subject,source_ids,detail. Subjects are normalized users, domains, or attachment filenames. Detail is your own nonempty explanation. Sort by rule, subject, then source IDs.
  • timeline.csv: id,timestamp,source,action,user,host,owner. Include windowed auth events and a windowed email row. Normalize UTC using isoformat() with Z, retaining fractional seconds; sort by actual time then ID. Auth action is its outcome, source is auth.jsonl, and owner comes from the asset or is blank. The email row ID is email: followed by its source ID, source is suspicious.eml, action is email_received, and user/host/owner are blank.
  • errors.csv: source,line,reason, sorted by source, numeric line, then reason. Always include CSV headers, even with zero rows.
  • evidence_manifest.json: algorithm is sha256; files is the sorted relative-path, byte-size and SHA-256 inventory of every original case file, including configuration, email and evidence files.
  • case_summary.json: case_id, accepted_events, window_events, findings, errors, sorted unique unknown_hosts, evidence_complete, and a nonempty limitations list. Unknown hosts come from windowed auth events. Evidence is complete only when there are no errors or unknown hosts. Explain that headers are unverified and findings do not establish compromise.

Use UTF-8, proper CSV quoting, and write mode. Return the same summary you save. Repeated successful runs replace prior reports, including when the new investigation has no findings. Do not alter the case files. Checks exercise changed policies, malformed and duplicate records, benign lookalikes, Unicode, missing inputs, and reruns as well as the supplied case.

Try an example.py
from pathlib import Path
import json
from case_tools import hash_inventory
print(json.dumps(hash_inventory(Path("case")), indent=2))
YOUR TURN

Put it into practice.

Implement investigate(case_dir="case", output_dir="output") and call it under the main guard. Follow the case and report contracts in the lesson: validate required inputs and policy, quarantine invalid/duplicate auth records, select the UTC window, detect repeated failures and exact email indicators, join asset context, and write all five reports. Return the saved summary. The supplied case_tools helpers are available. Keep original evidence unchanged; repeated successful runs replace reports.

See an explained solution
One way to solve it.py
import csv, json, re
from pathlib import Path
from collections import defaultdict
from datetime import timezone
from email import policy as email_policy
from email.parser import BytesParser
from email.utils import parsedate_to_datetime
from case_tools import parse_utc, inspect_url, attachment_manifest, hash_inventory

def load_configuration(root):
    policy = json.loads((root / "policy.json").read_text(encoding="utf-8"))
    if (not isinstance(policy, dict) or not isinstance(policy.get("case_id"), str) or not policy["case_id"].strip()
        or type(policy.get("threshold")) is not int or policy["threshold"] < 1
        or not all(isinstance(policy.get(k), str) for k in ("start", "end"))):
        raise ValueError("Invalid investigation policy")
    start, end = parse_utc(policy["start"]), parse_utc(policy["end"])
    if start >= end:
        raise ValueError("Policy end must follow start")
    indicators = json.loads((root / "indicators.json").read_text(encoding="utf-8"))
    if not isinstance(indicators, dict) or not all(isinstance(indicators.get(k), list) for k in ("domains", "sha256")):
        raise ValueError("Invalid indicator configuration")
    normalize = lambda value: value.strip().lower().rstrip(".")
    if not all(isinstance(v, str) and normalize(v) for v in indicators["domains"]):
        raise ValueError("Invalid domain indicator")
    if not all(isinstance(v, str) and re.fullmatch(r"[a-fA-F0-9]{64}", v) for v in indicators["sha256"]):
        raise ValueError("Invalid hash indicator")
    domains = {normalize(v) for v in indicators["domains"]}
    hashes = {v.lower() for v in indicators["sha256"]}
    assets = json.loads((root / "assets.json").read_text(encoding="utf-8"))
    if not isinstance(assets, list):
        raise ValueError("Assets must be a list")
    index = {}
    for asset in assets:
        if (not isinstance(asset, dict) or not all(isinstance(asset.get(k), str) and asset[k].strip() for k in ("host", "owner"))
            or type(asset.get("critical")) is not bool):
            raise ValueError("Invalid asset record")
        host = asset["host"].strip().casefold()
        if host in index:
            raise ValueError("Duplicate asset host")
        index[host] = asset
    return policy, start, end, domains, hashes, index

def read_auth(path):
    events, errors, seen = [], [], set()
    for number, line in enumerate(path.read_text(encoding="utf-8").splitlines(), 1):
        if not line.strip():
            continue
        try:
            event = json.loads(line)
        except json.JSONDecodeError:
            errors.append({"source":"auth.jsonl", "line":number, "reason":"invalid JSON"})
            continue
        try:
            if (not isinstance(event, dict)
                or not all(isinstance(event.get(k), str) and event[k].strip() for k in ("id", "timestamp", "user", "host", "outcome"))
                or event["outcome"] not in ("failure", "success")):
                raise ValueError("Invalid event")
            moment = parse_utc(event["timestamp"])
        except ValueError:
            errors.append({"source":"auth.jsonl", "line":number, "reason":"invalid event"})
            continue
        event_id = event["id"].strip()
        if event_id in seen:
            errors.append({"source":"auth.jsonl", "line":number, "reason":"duplicate id"})
            continue
        seen.add(event_id)
        events.append({**event, "id":event_id, "user":event["user"].strip().casefold(), "host":event["host"].strip().casefold(), "moment":moment})
    return events, errors

def write_csv(path, fields, rows):
    with path.open("w", encoding="utf-8", newline="") as handle:
        writer = csv.DictWriter(handle, fieldnames=fields)
        writer.writeheader()
        writer.writerows(rows)

def investigate(case_dir="case", output_dir="output"):
    root, output = Path(case_dir), Path(output_dir)
    if output.resolve().is_relative_to(root.resolve()):
        raise ValueError("Reports must be outside the evidence directory")
    for name in ("policy.json", "indicators.json", "assets.json", "auth.jsonl", "suspicious.eml"):
        if not (root / name).is_file():
            raise FileNotFoundError(name)
    if not (root / "evidence").is_dir():
        raise FileNotFoundError("evidence")
    settings, start, end, domains, hashes, assets = load_configuration(root)
    raw = (root / "suspicious.eml").read_bytes()
    message = BytesParser(policy=email_policy.default).parsebytes(raw)
    try:
        email_time = parsedate_to_datetime(str(message.get("Date", "")))
        if email_time.tzinfo is None:
            raise ValueError("Email date needs an offset")
        email_time = email_time.astimezone(timezone.utc)
    except (ValueError, TypeError) as error:
        raise ValueError("Email needs a valid aware Date header") from error
    message_id = str(message.get("Message-ID", "")).strip() or "email"
    events, errors = read_auth(root / "auth.jsonl")
    selected = [event for event in events if start <= event["moment"] < end]
    findings, timeline, failures = [], [], defaultdict(list)
    for event in selected:
        asset = assets.get(event["host"])
        timeline.append({"id":event["id"], "timestamp":event["moment"].isoformat().replace("+00:00", "Z"), "source":"auth.jsonl", "action":event["outcome"], "user":event["user"], "host":event["host"], "owner":asset["owner"] if asset else ""})
        if event["outcome"] == "failure":
            failures[event["user"]].append(event["id"])
    for user, ids in failures.items():
        if len(ids) >= settings["threshold"]:
            findings.append({"rule":"repeated-failures", "subject":user, "source_ids":"|".join(sorted(ids)), "detail":f"{len(ids)} failures in the selected window"})
    if start <= email_time < end:
        timeline.append({"id":"email:"+message_id, "timestamp":email_time.isoformat().replace("+00:00", "Z"), "source":"suspicious.eml", "action":"email_received", "user":"", "host":"", "owner":""})
        body = message.get_body(preferencelist=("plain",))
        matched = set()
        for value in re.findall(r'https?://[^\s<>"\x27]+', body.get_content() if body else "", flags=re.IGNORECASE):
            try:
                host = inspect_url(value.rstrip(".,;)"))["host"]
            except ValueError:
                errors.append({"source":"suspicious.eml", "line":0, "reason":"invalid URL"})
                continue
            if host in domains:
                matched.add(host)
        for host in matched:
            findings.append({"rule":"email-domain", "subject":host, "source_ids":message_id, "detail":"Exact domain indicator in a plain-text email URL"})
        for part in attachment_manifest(raw):
            if part["sha256"] in hashes:
                findings.append({"rule":"attachment-hash", "subject":part["filename"], "source_ids":message_id, "detail":"SHA-256 indicator: "+part["sha256"]})
    findings.sort(key=lambda row:(row["rule"], row["subject"], row["source_ids"]))
    timeline.sort(key=lambda row:(parse_utc(row["timestamp"]), row["id"]))
    errors.sort(key=lambda row:(row["source"], row["line"], row["reason"]))
    unknown = sorted({event["host"] for event in selected if event["host"] not in assets})
    summary = {"case_id":settings["case_id"], "accepted_events":len(events), "window_events":len(selected), "findings":len(findings), "errors":len(errors), "unknown_hosts":unknown, "evidence_complete":not errors and not unknown, "limitations":["Email headers are unverified observations.", "Indicator matches do not establish compromise."]}
    manifest = {"algorithm":"sha256", "files":hash_inventory(root)}
    output.mkdir(parents=True, exist_ok=True)
    write_csv(output / "findings.csv", ["rule","subject","source_ids","detail"], findings)
    write_csv(output / "timeline.csv", ["id","timestamp","source","action","user","host","owner"], timeline)
    write_csv(output / "errors.csv", ["source","line","reason"], errors)
    for name, data in [("evidence_manifest.json", manifest), ("case_summary.json", summary)]:
        (output / name).write_text(json.dumps(data, ensure_ascii=False, indent=2)+"\n", encoding="utf-8")
    return summary

if __name__ == "__main__":
    print(json.dumps(investigate(), ensure_ascii=False, indent=2))

The tool validates its configuration before output, separates collection errors from findings, and carries source IDs into every report. Exact URL hosts and decoded attachment hashes provide reproducible leads; original-byte manifests and explicit limitations keep the investigation reviewable. The checks vary the dataset and policy so a hard-coded report cannot pass.