01 / EVIDENCE AND SECURITY TEXT · LESSON 04

Match indicators precisely

Match complete normalised domain indicators without interpreting them as patterns.

16 min · learn at your pace

Indicators are values

A dot has meaning in a regular expression: it can match any character. An indicator such as login.example should not silently become a pattern that also matches loginXexample. Likewise, a substring match would include login.example.attacker.invalid, which is a different domain.

For complete lines, grep -F -x combines fixed text with whole-line matching. For a field in a table, an awk lookup set is often simpler. Array membership compares the complete key. Do not interpolate indicator text into a shell command or an awk program; read it as input data.

Normalisation is part of the rule

This lesson compares ASCII domain strings after lowercasing and removing one final dot. It does not extract hostnames from URLs, resolve DNS, convert internationalised names or match subdomains. Those are different operations with separate edge cases. The supplied indicators are fictional and do not cause any network activity.

Use FILENAME == ARGV[1] when loading the first file. The familiar NR == FNR shortcut misclassifies the next file when the first is empty. An empty indicator list must produce zero matches, not a new rule set assembled from the evidence itself. A match is a reason to investigate, not proof of compromise.

Try an example.sh
printf 'login.example\nloginXexample\n' | grep -F -x 'login.example'
YOUR TURN

Put it into practice.

Load indicators.txt (one ASCII domain per line; ignore blank lines and lines beginning #). Lowercase and remove one trailing dot from both indicators and domains in dns.tsv, whose header is id<TAB>domain. Print id<TAB>indicator followed by matching event IDs and normalised domains in DNS input order. Match the entire domain, never suffixes or regexes. Duplicate indicators must not duplicate findings; an empty list matches nothing.

See an explained solution
One way to solve it.sh
export LC_ALL=C
printf 'id\tindicator\n'
awk -F '\t' '
function norm(s){s=tolower(s);sub(/\.$/,"",s);return s}
FILENAME==ARGV[1] {if ($0!="" && $0!~/^#/) indicators[norm($0)]=1;next}
FNR>1 {d=norm($2);if (d in indicators) printf "%s\t%s\n",$1,d}
' indicators.txt dns.tsv

An associative lookup treats the indicator as literal data. Applying the same narrowly defined normalisation to both sources allows case and final-dot differences without accepting misleading suffixes or regex lookalikes.