Collect bounded background jobs
Run jobs in batches of two and associate each exit status with its own output.
Concurrency needs ownership
Launching a command with an ampersand starts it asynchronously. The parent receives a process ID through $!, but the job's exit status arrives later through wait. Keep the ID together with the logical job name; otherwise a failed task can be attributed to the wrong host or endpoint.
This lesson uses batches of at most two jobs. It is deliberately simpler than a continuously replenished worker pool and works on Bash 3.2, where wait -n is unavailable. After starting a batch, wait for its tracked IDs, collect every result and then launch the next batch. One failure must not discard a neighbour's successful output.
Separate output before combining it
Two jobs writing a shared report can interleave bytes. Give each job its own temporary file, then assemble the final report in the requested order. Completion order is not a stable report order; a fast failure might finish before an earlier successful task.
The supplied job IDs are unique safe filenames and the helper emits a single line. Those are explicit input guarantees, not a generic excuse to build paths from arbitrary user input. The trap removes this script's temporary directory after collection. For tools that can be interrupted while jobs are still running, also track and stop their children before removing workspace; the downloadable scanner demonstrates that lifecycle separately.
(printf 'background result\n';exit 7) > example-job.txt &
pid=$!
if wait "$pid";then status=0;else status=$?;fi
printf 'status=%s output=%s\n' "$status" "$(cat example-job.txt)"
Put it into practice.
Source lib/jobs.sh. job_ids.txt lists unique safe IDs; run_job(id) prints one line and returns its recorded status (9/missing if absent). Start at most two background jobs in each batch, capture each output separately, wait for every PID, and replace results.tsv with job<TAB>status<TAB>output in job_ids.txt order. Keep failed jobs and finish with status 0. An empty list produces only the header. Do not let one child overwrite another’s output; clean up your temporary directory.
See an explained solution
source lib/jobs.sh
work=$(mktemp -d .jobs.XXXXXX) || exit 2
trap 'rm -rf "$work"' EXIT
printf 'job\tstatus\toutput\n' > results.tsv
pids=();ids=()
collect() {
local i status
for ((i=0;i<${#pids[@]};i++));do
if wait "${pids[$i]}";then status=0;else status=$?;fi
printf '%s\t%s\t%s\n' "${ids[$i]}" "$status" "$(cat "$work/${ids[$i]}")" >> results.tsv
done
pids=();ids=()
}
while IFS= read -r id;do
run_job "$id" > "$work/$id" &
pids+=("$!");ids+=("$id")
if [ "${#pids[@]}" -eq 2 ];then collect;fi
done < job_ids.txt
collect
Each owned PID stays paired with its logical job and output file. Waiting for all jobs preserves failures as useful results, while collecting by input order gives deterministic reports despite asynchronous execution.