02 / MAKE EXECUTION DEPENDABLE · LESSON 06

Collect bounded background jobs

Run jobs in batches of two and associate each exit status with its own output.

22 min · learn at your pace

Concurrency needs ownership

Launching a command with an ampersand starts it asynchronously. The parent receives a process ID through $!, but the job's exit status arrives later through wait. Keep the ID together with the logical job name; otherwise a failed task can be attributed to the wrong host or endpoint.

This lesson uses batches of at most two jobs. It is deliberately simpler than a continuously replenished worker pool and works on Bash 3.2, where wait -n is unavailable. After starting a batch, wait for its tracked IDs, collect every result and then launch the next batch. One failure must not discard a neighbour's successful output.

Separate output before combining it

Two jobs writing a shared report can interleave bytes. Give each job its own temporary file, then assemble the final report in the requested order. Completion order is not a stable report order; a fast failure might finish before an earlier successful task.

The supplied job IDs are unique safe filenames and the helper emits a single line. Those are explicit input guarantees, not a generic excuse to build paths from arbitrary user input. The trap removes this script's temporary directory after collection. For tools that can be interrupted while jobs are still running, also track and stop their children before removing workspace; the downloadable scanner demonstrates that lifecycle separately.

Try an example.sh
(printf 'background result\n';exit 7) > example-job.txt &
pid=$!
if wait "$pid";then status=0;else status=$?;fi
printf 'status=%s output=%s\n' "$status" "$(cat example-job.txt)"
YOUR TURN

Put it into practice.

Source lib/jobs.sh. job_ids.txt lists unique safe IDs; run_job(id) prints one line and returns its recorded status (9/missing if absent). Start at most two background jobs in each batch, capture each output separately, wait for every PID, and replace results.tsv with job<TAB>status<TAB>output in job_ids.txt order. Keep failed jobs and finish with status 0. An empty list produces only the header. Do not let one child overwrite another’s output; clean up your temporary directory.

See an explained solution
One way to solve it.sh
source lib/jobs.sh
work=$(mktemp -d .jobs.XXXXXX) || exit 2
trap 'rm -rf "$work"' EXIT
printf 'job\tstatus\toutput\n' > results.tsv
pids=();ids=()
collect() {
 local i status
 for ((i=0;i<${#pids[@]};i++));do
  if wait "${pids[$i]}";then status=0;else status=$?;fi
  printf '%s\t%s\t%s\n' "${ids[$i]}" "$status" "$(cat "$work/${ids[$i]}")" >> results.tsv
 done
 pids=();ids=()
}
while IFS= read -r id;do
 run_job "$id" > "$work/$id" &
 pids+=("$!");ids+=("$id")
 if [ "${#pids[@]}" -eq 2 ];then collect;fi
done < job_ids.txt
collect

Each owned PID stays paired with its logical job and output file. Waiting for all jobs preserves failures as useful results, while collecting by input order gives deterministic reports despite asynchronous execution.