jobflow by rofl0r

cat things.list | jobflow -threads=8 -exec ./mytask {}

seq 100 | jobflow -threads=100 -exec echo {}

cat urls.txt | jobflow -threads=32 -exec wget {}

find . -name '*.bmp' | jobflow -threads=8 -exec bmp2jpeg {.}.bmp {.}.jpg

seq 100 | jobflow -skip 10 -count 10  # print lines 11 to 20

-skip N -threads N -resume -statefile=/tmp/state -delayedflush
-delayedspinup N -buffered -joinoutput -limits mem=16M,cpu=10
-eof=XXX
-exec ./mycommand {}

N=number of entries to skip

N=only process count lines (after skipping)

N=number of parallel processes to spawn

resume from last jobnumber stored in statefile

use XXX as the EOF marker on stdin
if the marker is encountered, behave as if stdin was closed
not compatible with pipe/bulk mode

XXX=filename
saves last launched jobnumber into a file

only write to statefile whenever all processes are busy,
and at program end

N=maximum amount of milliseconds
...to wait when spinning up a fresh set of processes
a random value between 0 and the chosen amount is used to delay initial
spinup.
this can be handy to circumvent an I/O lockdown because of a burst of
activity on program startup

store the stdout and stderr of launched processes into a temporary file
which will be printed after a process has finished.
this prevents mixing up of output of different processes.

if -buffered, write both stdout and stderr into the same file.
this saves the chronological order of the output, and the combined output
will only be printed to stdout.

do bulk copies with a buffer of N bytes. only usable in pipe mode.
this passes (almost) the entire buffer to the next scheduled job.
the passed buffer will be truncated to the last line break boundary,
so jobs always get entire lines to work with.
this option is useful when you have huge input files and relatively short
task runtimes. by using it, syscall overhead can be reduced to a minimum.
N must be a multiple of 4KB. the suffixes G/M/K are detected.
actual memory allocation will be twice the amount passed.
note that pipe buffer size is limited to 64K on linux, so anything higher
than that probably doesn't make sense.

sets the rlimit of the new created processes.
see "man setrlimit" for an explanation. the suffixes G/M/K are detected.

everything past -exec is treated as the command to execute on each line of
stdin received. the line can be passed as an argument using {}.
{.} passes everything before the last dot in a line as an argument.
it is possible to use multiple substitutions inside a single argument,
but currently only of one type.
if -exec is omitted, input will merely be dumped to stdout (like cat).

echo "CFLAGS=-O2 -g" > config.mak
make -j2