01 / EVIDENCE AND SECURITY TEXT · LESSON 02

Validate security records

Separate valid TSV records from rejected rows with physical line references.

16 min · learn at your pace

A tab is a boundary

Structured logs are useful only when their structure is understood. This exercise uses three tab-separated fields with no embedded tabs or newlines. It is not CSV: quoted commas and escaped line breaks need a real parser, not a quick change of separator.

awk -F '\t' splits on tabs and keeps empty fields. A Bash read loop with a whitespace separator can collapse adjacent separators, which hides missing data. Validate the number of fields before examining their contents. A row with an empty user is different from a row with too many fields.

Reject visibly

Keep accepted data in one file and rejection reasons in another. Physical line numbers include the header and allow a reviewer to locate a rejected record. Define a single precedence for errors so the same row gets the same explanation on every run.

Use anchored patterns for identifiers and exact comparisons for enumerated values. A result containing NOT_OK is not OK. Never treat malformed rows as successful authentication. Reports should be replaced on a rerun; appending old errors would count the same problem twice. The later capstone builds on this separation between usable evidence and evidence limitations.

Try an example.sh
awk -F '\t' 'NR > 1 {printf "line=%d fields=%d\n", NR, NF}' auth.tsv
YOUR TURN

Put it into practice.

Validate auth.tsv after its supplied header id<TAB>user<TAB>result. Write valid.tsv with that header and accepted rows in input order. Write errors.tsv with header line<TAB>reason. Reject in this order: NF != 3 as fields; an ID not matching ^[A-Za-z0-9_-]+$ as id; an empty user as user; any result other than FAIL or OK as result. Use physical line numbers. Blank records are invalid. Replace reports on each run.

See an explained solution
One way to solve it.sh
export LC_ALL=C
printf 'id\tuser\tresult\n' > valid.tsv
printf 'line\treason\n' > errors.tsv
awk -F '\t' 'NR>1 {
    reason=""
    if (NF!=3) reason="fields"
    else if ($1 !~ /^[A-Za-z0-9_-]+$/) reason="id"
    else if ($2=="") reason="user"
    else if ($3!="FAIL" && $3!="OK") reason="result"
    if (reason!="") printf "%d\t%s\n", NR,reason >> "errors.tsv"
    else print $0 >> "valid.tsv"
}' auth.tsv

The field-count check prevents later checks from interpreting missing columns. The ordered validation rules preserve a reproducible rejection reason, and headers are rewritten so repeated runs do not duplicate results.