Capstone: investigate operational logs
Combine portable Bash and Unix tools into a reproducible operational-log investigation.
Turn a question into a repeatable tool
An operations team has two fictional log exports. They want to review warning and error events without losing the source of each finding. Your tool should produce the same reports whenever it receives the same evidence. A deterministic report makes later review and comparison much easier.
Start with the contract in Put it into practice. It defines file order, allowed fields, UTC dates, duplicate handling and report ordering. A suspicious or severe event is a reason to investigate; it does not establish the underlying cause by itself. Preserve the original messages and source line numbers so another person can follow your reasoning.
Validate before grouping
Check required files and headers before processing records. A malformed individual record belongs in errors.tsv; it should not discard the rest of the evidence. Validate all fields before recording an ID as seen. Otherwise an invalid early row could suppress a later valid event. Only accepted WARN and ERROR records belong in the findings and counts reports.
Fixed-width UTC timestamps can be sorted as text after their calendar values have been validated. Do not compare arbitrary timestamp strings this way. Our explicit 2000–2099 range makes leap-year validation manageable without GNU-only date parsing.
Publish a coherent result
Use a temporary directory and an EXIT trap. Generate complete output files there, then move them into the report directory. Replace reports on each successful run. Avoid in-place edits to evidence, and keep stderr for failures rather than mixing diagnostic text into TSV data.
Break the work into small functions and awk rules. Test a header-only dataset, an invalid row, duplicates across both files, a missing input and a path containing spaces. The automatic checks also rerun your tool and verify that the evidence stays unchanged. There is no single required implementation: a different Bash 3.2-compatible solution that meets the behavior is welcome.
awk -F '\t' 'FNR > 1 && ($4 == "WARN" || $4 == "ERROR") {printf "%s\t%s\n", $1, $5}' evidence/application.tsv
Put it into practice.
Build `bash main.sh [input_directory] [output_directory]`, defaulting to `evidence` and `reports`, with at most two arguments. Use a separate output directory. Required inputs are `application.tsv` and `authentication.tsv`; both must start with the exact header `event_id<TAB>timestamp<TAB>service<TAB>severity<TAB>message`. Missing inputs or an invalid header must fail clearly on stderr with nonzero status.
Read application.tsv before authentication.tsv and retain line order within each. Each data record has exactly five tab-separated fields with no embedded tabs/newlines: event_id is 1–40 ASCII letters/digits/underscore/hyphen; timestamp is a real UTC calendar date from 2000–2099 in YYYY-MM-DDTHH:MM:SSZ form (seconds 00–59); service is 1–32 lowercase ASCII letters/digits/hyphens starting with a letter; severity is INFO, WARN or ERROR; message is 1–240 UTF-8 bytes with no ASCII control characters. Do not trim or normalize fields. Reject invalid records using the first applicable reason in this order: fields, event_id, timestamp, service, severity, message. After validation, keep the first valid occurrence of each event_id, including INFO records; reject later valid occurrences as duplicate. Invalid occurrences do not reserve an ID.
Produce four newline-terminated reports. events.tsv has header `event_id<TAB>timestamp<TAB>service<TAB>severity<TAB>message<TAB>source<TAB>line`, and only WARN/ERROR records, sorted by timestamp then event_id with LC_ALL=C. source is the input basename and line is its 1-based physical line number including the header. counts.tsv has header `service<TAB>severity<TAB>count` and counts selected records, sorted by service then severity with LC_ALL=C. errors.tsv has header `source<TAB>line<TAB>reason` and rejected rows in processing order. summary.txt contains exactly `inputs=2`, `accepted=N`, `selected=N`, `rejected=N`, one per line; accepted counts unique valid records of every severity.
Header-only inputs are valid empty datasets. Successful runs print no terminal output, leave evidence unchanged and replace previous reports. Spaces/Unicode in directory names and messages must work.
See an explained solution
#!/usr/bin/env bash
export LC_ALL=C
fail() { printf '%s\n' "$1" >&2; exit 1; }
[ "$#" -le 2 ] || fail 'Usage: bash main.sh [input_directory] [output_directory]'
input=${1:-evidence}
output=${2:-reports}
# Prefix relative paths so an initial dash cannot become an option.
case "$input" in /*) ;; *) input=./$input ;; esac
case "$output" in /*) ;; *) output=./$output ;; esac
for source in application.tsv authentication.tsv; do
[ -f "$input/$source" ] && [ -r "$input/$source" ] || fail "Missing input: $source"
IFS= read -r header < "$input/$source" || fail "Invalid header: $source"
expected=$(printf 'event_id\ttimestamp\tservice\tseverity\tmessage')
[ "$header" = "$expected" ] || fail "Invalid header: $source"
done
mkdir -p "$output" || fail 'Cannot create output directory'
input=$(cd "$input" && pwd -P) || fail 'Cannot resolve input directory'
output=$(cd "$output" && pwd -P) || fail 'Cannot resolve output directory'
[ "$input" != "$output" ] || fail 'Use a separate output directory'
work=$(mktemp -d "${TMPDIR:-/tmp}/rocks-logs.XXXXXX") || fail 'Cannot create temporary directory'
cleanup() { rm -rf "$work"; }
trap cleanup EXIT
trap 'exit 1' HUP INT TERM
export REPORT_STAGE="$work"
: > "$work/events.raw"
: > "$work/counts.raw"
printf 'source\tline\treason\n' > "$work/errors.tsv"
awk -F '\t' '
function valid_time(s, y,m,d,h,mi,se,days) {
if (s !~ /^20[0-9][0-9]-[0-9][0-9]-[0-9][0-9]T[0-9][0-9]:[0-9][0-9]:[0-9][0-9]Z$/) return 0
y=substr(s,1,4)+0; m=substr(s,6,2)+0; d=substr(s,9,2)+0
h=substr(s,12,2)+0; mi=substr(s,15,2)+0; se=substr(s,18,2)+0
if (m<1 || m>12 || d<1 || h>23 || mi>59 || se>59) return 0
days=31
if (m==4 || m==6 || m==9 || m==11) days=30
if (m==2) days=(y%4==0 ? 29 : 28)
return d<=days
}
BEGIN { stage=ENVIRON["REPORT_STAGE"]; accepted=selected=rejected=0 }
FNR==1 { next }
{
source=FILENAME; sub(/^.*\//,"",source)
reason=""
if (NF!=5) reason="fields"
else if ($1 !~ /^[A-Za-z0-9_-]+$/ || length($1)>40) reason="event_id"
else if (!valid_time($2)) reason="timestamp"
else if ($3 !~ /^[a-z][a-z0-9-]*$/ || length($3)>32) reason="service"
else if ($4!="INFO" && $4!="WARN" && $4!="ERROR") reason="severity"
else if (length($5)<1 || length($5)>240 || $5 ~ /[[:cntrl:]]/) reason="message"
else if ($1 in seen) reason="duplicate"
if (reason!="") {
printf "%s\t%d\t%s\n",source,FNR,reason >> (stage "/errors.tsv")
rejected++; next
}
seen[$1]=1; accepted++
if ($4=="WARN" || $4=="ERROR") {
printf "%s\t%s\t%d\n",$0,source,FNR >> (stage "/events.raw")
counts[$3 SUBSEP $4]++; selected++
}
}
END {
for (key in counts) {
split(key,parts,SUBSEP)
printf "%s\t%s\t%d\n",parts[1],parts[2],counts[key] >> (stage "/counts.raw")
}
printf "inputs=2\naccepted=%d\nselected=%d\nrejected=%d\n",accepted,selected,rejected > (stage "/summary.txt")
}' "$input/application.tsv" "$input/authentication.tsv" || fail 'Could not process logs'
printf 'event_id\ttimestamp\tservice\tseverity\tmessage\tsource\tline\n' > "$work/events.tsv"
tab=$(printf '\t')
sort -t "$tab" -k2,2 -k1,1 "$work/events.raw" >> "$work/events.tsv" || fail 'Could not sort events'
printf 'service\tseverity\tcount\n' > "$work/counts.tsv"
sort -t "$tab" -k1,1 -k2,2 "$work/counts.raw" >> "$work/counts.tsv" || fail 'Could not sort counts'
for report in events.tsv counts.tsv errors.tsv summary.txt; do
mv "$work/$report" "$output/$report" || fail "Could not publish $report"
done
The reference script validates inputs up front, uses an isolated temporary directory, and processes records in the documented order. Its awk program validates real calendar dates, reserves only valid IDs and carries source evidence into each selected row. Sorting occurs after extraction; reports are moved into place only after processing succeeds. The EXIT trap removes temporary work on success or failure.